Computer display device mounted on eyeglasses

ABSTRACT

When an imaging apparatus ( 30 ) captures an image of a finger performing an operation on a visual confirmation screen (S 2 ), an operation determining unit ( 73 ) of a computer display device ( 1 ) mounted on eyeglasses ( 10 ) determines the contents of the operation performed by the finger based on image data obtained by the imaging apparatus ( 30 ). A positional data generating unit ( 74 ) generates positional data of the finger based on the image data. Based on data concerning the determined contents of the operation performed by the finger, the generated positional data of the finger, and reference data concerning the visual confirmation screen stored in a storage unit ( 90 ), an input control unit ( 76 ) determines the contents of an inputted instruction corresponding to the operation performed by the finger, and controls an original screen to be displayed on an display apparatus ( 20 ) in accordance with the determined contents of the inputted instruction.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Japanese patent applicationserial number 2014-152019 filed on Jul. 25, 2014 and to Japanese patentapplication serial number 2015-110240 filed on May 29, 2015, thecontents of both of which are incorporated fully herein.

TECHNICAL FIELD

The present invention generally relates to wearable computers, such ascomputer display devices mountable on eyeglasses (eyeglass-type computerdevices), “smartglasses”, head-mounted displays, optical head-mounteddisplays, etc.

BACKGROUND ART

In recent years, wearable computers, such as computer display devicesmounted on eyeglasses, including Google Glass^(Tm) made by Google Inc.,have been developed and marketed; see also Japanese Patent ApplicationLaid-open No. H11-98227. In a Google Glass device, an ultra-smalldisplay that uses a prism is arranged in front of the right eyeglass ofthe eyeglass frame (main body), enabling the Google Glass user tosimultaneously view a virtual screen displayed on the display togetherwith the real surroundings of the user. For example, by wearing thistype of computer display device, the user can walk around town whileviewing information, such as a map of the neighborhood, displayed on avirtual screen that appears to be floating in the peripheral vision ofthe user's right eye. In addition, for example, stores, restaurants,etc. in the vicinity of the user can display advertisements on thescreen for the user.

SUMMARY OF THE INVENTION

With currently developed and marketed computer display devices mountedon eyeglasses (smartglasses), however, in order to operate a displayed(virtual) screen, the user issues a voice command or taps a touchpadprovided at a base (side) portion of the eyeglass frame. However, wheninputting characters into a visual (virtual) confirmation screen using,for example, a voice command, not everyone can achieve accuratecharacter input because languages have a large number of homonyms andthus vocalization results in individual pronunciation differences thatmay not be accurately understood by the speech recognition software. Inaddition, the touchpad provided on the eyeglass frame is not largeenough to enable manual character input. Therefore, known computerdisplay devices mounted on eyeglasses s suffer from the problem thatperforming character input for an e-mail, etc. using a virtual screendisplayed in the user's field of vision is difficult.

An object of the present teachings is to provide computer displaydevices mounted on eyeglasses, smartglasses, wearable computers, etc.that enable operations, such as character input on a virtual screendisplayed in the user's field of vision, to be performed easily andaccurately.

A first aspect of the present teachings is provided in appended claim 1.

With such a device, when the user performs an operation using a fingeror an input instructing tool on the visual (virtual) confirmationscreen, the input control unit recognizes the contents of the inputtedinstructions corresponding to the operation performed by the finger orthe input instructing tool based on (i) data concerning the contents ofthe operation performed by the finger or the input instructing tool asdetermined by the operation determining unit, (ii) the positional dataof the finger or the input instructing tool generated by the positionaldata generating unit, and (iii) the reference data concerning the visualconfirmation screen stored in a storage unit, and controls the originalscreen to be displayed on the display apparatus in accordance with therecognized contents of the inputted instruction. Therefore, byperforming, on a visual confirmation screen that appears as though it isfloating in midair, an operation (a virtual manual operation) similar toa manual (touch) operation performed on a screen displayed on aconventional touch panel or touch screen (e.g., a smartphone or a tabletcomputer), the user can input an instruction corresponding to the manualoperation. Consequently, by using the device according to appended claim1, the user can easily and accurately perform character input operationsand various screen operations, including enlargement and reduction, byoperating (manipulating) a visual (virtual) confirmation screen in amanner to similar a conventional smartphone display or a tablet screen.

In addition or in the alternative, a second aspect of the presentteachings is provided in appended claim 5.

With such a device, when the user performs an operation using a fingeror an input instructing tool on the visual confirmation screen and thedistance determining unit determines that the finger or the inputinstructing tool is at a position within a range of an approximatelyconstant distance from the reference screen, the input control unitrecognizes the contents of the inputted instruction corresponding to theoperation performed by the finger or the input instructing tool based on(i) the data concerning the contents of the operation performed by thefinger or the input instructing tool as determined by the operationdetermining unit, (ii) the positional data of the finger or the inputinstructing tool generated by the positional data generating unit basedon image data used by the operation determining unit to make thedetermination, (iii) the distance data associated with the image dataused by the operation determining unit to make the determination, and(iv) the reference data that specifies a reference screen that is storedin a storage unit and corresponds to the visual confirmation screen, andcontrols an original screen to be displayed on the display apparatus inaccordance with the recognized contents of the inputted instruction.Therefore, by performing, on the visual (virtual) confirmation screenthat appears to be floating in midair, a manual (virtual touch)operation similar to a manual operation performed on a screen displayedon a conventional touch panel or touch screen (e.g., a smartphone ortablet computer), the user can input an instruction corresponding to themanual operation. Consequently, by using the device according to theappended claim 5, the user can easily and accurately perform characterinput operations and various screen operations, including enlargementand reduction, by operating a visual (virtual) confirmation screen in amanner similar to a conventional smartphone display or a tablet screen.In addition, since the reference data generating unit generates data,which specifies the reference screen that corresponds to the visualconfirmation screen in a three-dimensional space, as reference data,even if the user has a habit of, when manually operating (virtuallymanipulating) the visual confirmation screen, operating (touching) atpositions on a near side with respect to the two left-side corners ofthe visual confirmation screen and operating (touching) at positions ona far side with respect to the two right-side corners of the visualconfirmation screen, reference data conforming to such a habit can begenerated.

With computer display devices mounted on eyeglasses according to thepresent teachings, various screen operations such as character input andenlargement and reduction can be easily and accurately performed on avisual confirmation screen displayed in the user's field of vision.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic plan view of a computer display device mounted oneyeglasses according to a first embodiment of the present teachings, andFIG. 1B is a schematic right side view of the same;

FIG. 2 is a schematic perspective view of the computer display deviceaccording to the first embodiment;

FIG. 3 is a schematic block diagram of the computer display deviceaccording to the first embodiment;

FIG. 4 is a diagram showing an example of a character input screen;

FIG. 5 is a diagram showing an example of a search screen that isdisplayed on a character input screen;

FIGS. 6A and 6B show diagrams for explaining alternate modes forperforming a touch operation on a visual confirmation screen;

FIGS. 7A and 7B show diagrams for explaining alternate modes forperforming a touch operation performed on a visual confirmation screen;

FIG. 8 is a flow chart that explains a procedure for setting referencedata in the computer display device according to the first embodiment;

FIG. 9A-9C are diagrams that show an example of an original screendisplayed during the reference data setting procedure;

FIG. 10 is a flow chart that explains a character input procedure in thecomputer display device according to the first embodiment;

FIG. 11 is a flow chart that explains a screen display procedure in thecomputer display device according to the first embodiment;

FIG. 12 is a schematic block diagram of a computer display devicemounted on eyeglasses according to a second embodiment of the presentteachings;

FIG. 13 is a schematic block diagram of a computer display devicemounted on eyeglasses according to a first modification of the secondembodiment of the present teachings;

FIG. 14 is a diagram that explains a process performed by a deviationcorrecting unit to convert an X coordinate of positional data into an Xcoordinate of positional data on a reference screen K according to thefirst modification of the second embodiment;

FIG. 15 is a diagram for explaining a process performed by a deviationcorrecting unit to convert a Y coordinate of positional data into a Ycoordinate of positional data on the reference screen K according to thefirst modification of the second embodiment;

FIG. 16 is a schematic block diagram of a computer display devicemounted on eyeglasses according to a third embodiment of the presentteachings;

FIGS. 17A and 17B are diagrams showing an example of an original screenfor setting reference data displayed during the reference data settingprocedure according to the third embodiment;

FIG. 18 is a flow chart that explains a character input procedure in thecomputer display device according to the third embodiment;

FIG. 19 is a flow chart that explains a screen display procedure in thecomputer display device according to the third embodiment;

FIG. 20 is a diagram showing an embodiment in which a remote controlscreen of an air conditioner is used as the original screen and a userperforms an operation on a visual (virtual) confirmation screencorresponding to the remote control screen;

FIG. 21A is a schematic plan view of a computer display device mountedon eyeglasses according to a modification of the first embodiment of thepresent teachings, and

FIG. 21B is a schematic right side view of the same;

FIG. 22A is a schematic perspective view of a computer display devicemounted on eyeglasses according to a modification of the firstembodiment, and FIG. 22B is a schematic view that explains an embodimentin which the original screen is projected on a display apparatus;

FIG. 23 is a schematic plan view of a computer display device mounted oneyeglasses according to a second modification of the second embodimentof the present teachings and is a diagram that explains a processperformed by a deviation correcting unit to convert an X coordinate ofpositional data into an X coordinate of positional data on a referencescreen K; and

FIG. 24 is a schematic right side plan view of the computer displaydevice according to the second modification of the second embodiment ofthe present teachings and is a diagram that explains a process performedby a deviation correcting unit to convert a Y coordinate of positionaldata into a Y coordinate of positional data on the reference screen K.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, various embodiments of the present teachings will bedescribed with reference to the drawings.

First Embodiment

A computer display device mounted on eyeglasses according to a firstembodiment of the present teachings will be described first with respectto FIGS. 1-3.

The computer display device according to the first embodiment may begenerally constructed in the manner, for example, of a Google Glass™manufactured by Google Inc., which the user can wear like eyeglasses(spectacles). As shown in FIGS. 1 to 3, the computer display device 1includes an eyeglass main body (eyeglass frame) 10 having two lenssections (lenses) 11, a display apparatus 20 provided on the eyeglassmain body 10, an imaging apparatus 30 for capturing (sensing, recording,detecting, etc.) an image in front of the user, a touchpad unit 40, amicrophone unit 50, a speaker unit 60, a control unit 70, acommunicating unit 80, and a storage unit (also referred to as storageand/or memory) 90. It is noted that, in the further description of theembodiments, the computer display device 1 mounted on eyeglasses mayalso be referred to simply as a computer display device 1.

As shown in FIG. 2, the eyeglass main body 10 may comprise aconventional eyeglass frame having two lens sections (lens holders orrims) 11 and two temples (arms) attached, e.g., pivotably attached,thereto. Lenses mounted in the lens sections (lens holders or rims) 11may be convex lenses or concave lenses for correcting eyesight or may belenses made of ordinary glass, plastic, or the like without an eyesightcorrecting function. Alternatively, lenses may be omitted.

The display apparatus 20 may be a transmissive prism display. Forexample, the display apparatus 20 may comprise a small projectorincluding a liquid crystal panel (display device) or the like, anoptical system, and a half mirror. In such an embodiment, as shown inFIGS. 1 and 2, the half mirror is embedded in a prism arranged on theeyeglass main body 10 (e.g. on the frame) in front of the lens section11 of the right eye. An image or a video displayed on the liquid crystalpanel is projected onto the half mirror via the optical system. Inactual practice, a very small screen is displayed on the displayapparatus 20 (half mirror). By using the computer display device 1, auser can view a translucent screen (virtual screen/display) that is animage of the very small screen displayed on the display apparatus 20(half mirror) as though the translucent screen is floating in midair.From the perspective of the user, the translucent screen that appears tobe floating corresponds to (has the same apparent size as) a 25 inch(63.5 cm) screen when viewed from a distance of 8 feet (2.44 m).Although the present embodiment includes a translucent screen thatappears to be floating, generally speaking, the screen need not betranslucent. In addition, although the floating screen of the firstembodiment is displayed at an upper right position in the user's fieldof vision as shown in FIGS. 1 and 2, generally speaking, the screen maybe displayed at a center position, an upper left position, a lower rightposition, or the like of the user's field of vision. In the followingdescription, the very small screen displayed on the display apparatus 20(half mirror) will also be referred to as the “original screen” and thescreen that appears to the user of the computer display device 1 to befloating in midair will also be referred to as the “visual confirmationscreen”. The visual confirmation screen is a virtual screen that appearsto be projected in front of the user and is viewable by the user to readinformation, as well as to manually input instructions (commands) in themanner described below.

For example, various screens (screen images or display images)containing a character input screen may be displayed on the displayapparatus 20. FIG. 4 is a diagram showing an example of arepresentative, non-limiting character input screen. As shown in FIG. 4,the character input screen 200 includes a keyboard image 210 and adisplay area 220 for displaying the inputted characters, etc. Thekeyboard image 210 includes a plurality of character key imagesrespectively associated with alpha-numeric characters, punctuationmarks, symbols, etc. and a plurality of function key images to whichspecific functions are assigned. In the example shown in FIG. 4, theQWERTY keyboard layout is adopted as the layout of the character keyimages in the keyboard image 210. Alternatively, the keyboard image 210may be, e.g., a keyboard image in the layout of the Japanese syllabary,a keyboard image depicting characters for a non-English language (or alanguage that uses alpha-numeric characters other than Roman lettersand/or Arabic numerals), a ten-key image, a key image similar to a keylayout of a mobile phone, or the like. In addition, a search screen maybe displayed in the display area 220. FIG. 5 shows an example of asearch screen 221 that may be displayed on the character input screen200. The search screen 221 is designed for searching/selecting Internetweb sites and includes a keyword input section 2211 and a search resultdisplay section 2212 that displays the search results. When the user isviewing the character input screen 200 as the visual confirmation screenS, the user can input a keyword into the keyword input section 2211using key images (character keys) of the keyboard image 210.

In the first embodiment, the user can give (input) various instructionsto (into) the control unit 70 by using a finger to perform a touch(manual) operation (virtual touch) on the visual confirmation screen S.As will be described below, the control unit 70 recognizes (detects,determines, analyzes) the contents (specifics, intention) of theinstruction and, in accordance with the recognized (detected,determined, analyzed) contents of the instruction, controls (selects,generates) the original screen M that is displayed on the displayapparatus 20. In this embodiment, it is assumed that the touch operationencompasses various types of manually-performed operations including,without limitation, a tapping operation, a double-tapping operation, along-tapping operation, a dragging operation, a flicking operation, apinching-in operation, and a pinching-out operation, in a similar mannerto touch operations performed on conventional touch panels or touchscreens, e.g., smartphones or tablet computers.

As shown in FIGS. 1 and 2, the imaging apparatus 30 is provided(disposed) on the frame or, e.g., on one of the temples, of the eyeglassmain body 10 adjacent to the display apparatus 20. As shown in FIG. 3,the imaging apparatus 30 includes a camera unit 31, an image processingunit 32, and a camera control unit 33. The camera unit 31 includes alens and an imaging element, such as a charge-coupled device (CCD) or aCMOS active-pixel sensor, e.g., a digital camera. Based on image dataobtained by performing an imaging operation using the camera unit 31,the image processing unit 32 performs, e.g., one or more of: acorrection process of color or gradation of the captured image, imageprocessing such as compression of the image data, etc. The cameracontrol unit 33 controls the image processing unit 32 and controls theexchange (transfer) of image data with (to) the control unit 70. It isnoted that, although the present embodiment provides the imageprocessing unit 32 in the imaging apparatus 30, the image processingunit 32 may alternatively be provided in the control unit 70.

In addition, the imaging apparatus 30 may be designed to image orcapture only a part of the visual field or field of vision, orapproximately the entire visual field or field of vision, of the user asthe imaging range of the imaging apparatus 30. In particular, in thefirst embodiment, the imaging apparatus 30 is configured such that asubject (object) at the position (in the plane) of the visualconfirmation screen S as recognized by the user or, specifically theposition of the user's finger when the user reaches out with his/herhand to virtually touch the visual confirmation screen S and which(position or plane) is separated from the imaging apparatus 30 by anapproximately constant distance along a depth direction, is brought intofocus. In addition, the in-focus range (depth of field) is limited to anarrow range. For example, the in-focus position (depth) is set to aposition (depth) that is separated by approximately 40 cm from theimaging apparatus 30; the depth of field thereof is a range ofapproximately 5 cm around the in-focus position (depth). However, in thefirst embodiment, it is important to note that the imaging apparatus 30limits the in-focus range to the narrow range only when manualoperations for reference data setting, character input, and screendisplay are being performed, as will be discussed below. The in-focusrange is not limited to a narrow range when normal photography isperformed using the camera and in other situations. Moreover, theimaging apparatus 30, for example, may be designed to be capable ofswitching among (between) the in-focus positions by manually changingsettings using a distance ring (focus ring) in a same way as aconventional camera.

As was noted above, in the first embodiment, the in-focus position inthe imaging apparatus 30 is set to the position (depth) of the visualconfirmation screen S as recognized by the user. Therefore, when theuser is using a finger to (virtually) perform a manual operation on thevisual confirmation screen S, the imaging apparatus 30 is designed tocapture (detect, sense, analyze, etc.) an image of the finger performingthe operation in the state where the finger is in focus. Image dataobtained by the imaging operation performed by the imaging apparatus 30is sent (transmitted) to the control unit 70 and is stored in thestorage unit 90 by the control unit 70. In addition, the imagingapparatus 30 according to the first embodiment is equipped with astill-image photographing function and a moving-image photographing(video recording) function. The control unit 70 can thus acquire stillimage data or moving image data as necessary.

As shown in FIG. 3, the frame or temple of the eyeglass main body 10 maybe provided with the touchpad unit 40, the microphone unit 50, a boneconduction speaker (transmission) unit (bone conduction headset) 60,various sensor units, a battery unit, and the like. It should be notedthat these units have been omitted in FIGS. 1 and 2 for the sake ofbrevity and clarity. The touchpad unit 40 is used by the user to issue(input) various instructions (commands) to the control unit 70 byperforming touch (manual) operations on (physically manipulating) thetouchpad unit 40. The microphone unit 50 is designed for inputting theuser's voice (converting audio voice commands into digital instructions)in order to operate the eyeglass main body 10 (control unit 70) by voiceinstructions (voice commands). Voice information input from (via) themicrophone unit 50 is sent (transmitted) to the control unit 70 so thatthe digitized voice information is analyzed by the control unit 70. Inaddition, the speaker unit 60 is preferably designed for conveying voiceor audio information (audible sounds) to the user, e.g., using bonevibrations or conduction. However, the speaker unit 60 may comprise aconventional speaker, an earphone, a headphone, etc. instead of aspeaker that conveys voice information (audible sounds) to the userusing bone vibrations. Moreover, since the various sensor units, thebattery unit, and the like are not particularly pertinent to the presentteachings, a detailed description of these units may be omitted in thepresent embodiment. Conventional technology known to the person skilledin the art may be utilized to implement these units.

The control unit 70 preferably includes a central processing unit (CPU),etc. and controls the entire computer display device 1. For example, thecontrol unit 70 controls (selects, generates) the display of theoriginal screen M on the display apparatus 20 and controls the imagingoperation performed by the imaging apparatus 30. In addition, when thetouchpad unit 40 is manually operated, the control unit 70 recognizesthe contents of the manually-inputted instruction and executes one ormore processes in accordance with the recognized contents (inputtedinstructions). Similarly, when a voice command is input from (via) themicrophone unit 50, the control unit 70 recognizes the contents of theinputted voice information (command) and executes one or more processesin accordance with the recognized contents (inputted instructions).Furthermore, the control unit 70 controls (generates) the voiceinformation or audible sounds emitted by the speaker unit 60. Forexample, as shown in FIG. 3, the control unit 70 may include a displaycontrol unit 71, an image data extracting unit 72, an operationdetermining unit 73, a positional data generating unit 74, a referencedata generating unit 75, and an input control unit 76. As used anywherein the present description, any feature described as a “unit” or an“apparatus” may be embodied, e.g., as discrete physical units, asconceptual functional units, e.g., of software code (an operatingprogram) stored in the storage unit (memory) 90 that is executed by amicroprocessor, and/or as a hybrid hardware/firmware structure. Forexample, the physical form of units 71-76 (as well as other “units”disclosed herein) is not particularly limited in the present teachings.Furthermore, two or more “units” may be integrated together into asingle physical circuit structure, such as a CPU that is controlled bydifferent sets of programming code (stored instructions) capable ofperforming the particular functions when executed by a processor, suchas a microprocessor.

When the user issues a voice instruction (command) using the microphoneunit 50 or inputs a keyboard instruction by operating (manipulating) thetouchpad unit 40, the display control unit 71 selects (generates) thecontents of the original screen M to be displayed on the displayapparatus 20 and controls the display of the selected original screen Min accordance with contents of the inputted instruction. Accordingly,the original screen M instructed (requested) by the user is displayed onthe display apparatus 20 and the user can view the visual confirmationscreen S, which corresponds to the original screen M.

When an image of a subject (object) in focus is captured by the imagingapparatus 30 in accordance with a finger operation performed by the useron the visual confirmation screen S, the image data extracting unit 72determines whether or not the subject (object) is the finger based onthe image data obtained by the imaging and then extracts image data inwhich the finger is present. Known image (object) recognition methods orimage recognition (object recognition) software may be used to determinewhether or not the subject (object) is a finger. In the firstembodiment, since the depth of field of the imaging apparatus 30 islimited to a narrow range, when the subject (object) is determined to bea finger, it is considered/deemed that the finger is at a positionseparated (spaced) from the imaging apparatus 30 by an approximatelyconstant distance along the depth direction. In this manner, the imagedata extracting unit 72 extracts image data of the finger that is at aposition separated (spaced) from the imaging apparatus 30 by anapproximately constant distance along the depth direction. In addition,the operation determining unit 73, the positional data generating unit74, and the reference data generating unit 75 are configured to performvarious processes based on the image data extracted by the image dataextracting unit 72, as will be discussed below.

When the imaging apparatus 30 captures an image of the finger used bythe user to perform an operation on the visual confirmation screen S,the operation determining unit 73 determines the contents of the (touch)operation using the finger based on the image data obtained by theimaging operation and extracted by the image data extracting unit 72.Accordingly, the operation determining unit 73 can recognize that theoperation performed by the finger is, e.g., any one of a tappingoperation, a double-tapping operation, a long-tapping operation, etc.Data concerning the contents of the recognized operation performed bythe finger is stored in the storage unit 90.

When the imaging apparatus 30 captures an image of the finger used bythe user to perform the (touch) operation on the visual confirmationscreen S, the positional data generating unit 74 generates positionaldata of the finger (fingertip) in the imaging range of the imagingapparatus 30 based on the image data obtained by the imaging operationand extracted by the image data extracting unit 72. As shown in FIG. 2,the present embodiment utilizes an XY coordinate system within theimaging range of the imaging apparatus 30, wherein the horizontaldirection is the X axis direction and the vertical direction is the Yaxis direction. The origin of the XY coordinate system may be, forexample, a lower left point in the imaging range. The positional datagenerating unit 74 acquires positional data of the finger in this XYcoordinate system. Moreover, when it is necessary to obtainthree-dimensional positional data, the depth direction is the Z axisdirection, which is perpendicular to the XY coordinate system, and the Zaxis direction is used to constitute or define an XYZ coordinate system.

When the user performs an operation using a finger at one or a pluralityof prescribed (predetermined) positions on the visual confirmationscreen S, the reference data generating unit 75 generates dataconcerning the visual confirmation screen S using the positional data ofthe finger generated by the positional data generating unit 74 based onthe image data for which the operation determining unit 73 hasdetermined that an operation at each prescribed position is a prescribed(predetermined) operation. The generated data concerning the visualconfirmation screen S is stored in the storage unit 90 as referencedata. For this purpose, data that enables the position and the size ofthe visual confirmation screen S to be specified (set) is used as thereference data. For example, when the user performs an operation using afinger with respect to the four corners of an outer frame (periphery) ofthe visual confirmation screen S, as will be further discussed below,the positional data of the finger at each position of the four cornerscan be used as the reference data. In this case, since image dataextracted by the image data extracting unit 72 represents a capturedimage of the finger at a position separated (spaced) by an approximatelyconstant position along the Z axis direction from the imaging apparatus30, it is considered/deemed that the positional data of the finger ateach position of the four corners represents the positional informationof the finger on a plane parallel to an XY plane (approximately parallelto the body of the user) at a position separated (spaced) from theimaging apparatus 30 by the approximately constant distance along the Zaxis direction. In addition, when the user performs an operation using afinger with respect to one of the four corners of the outer frame(periphery) of the visual confirmation screen S, the positional data ofthe finger at the one corner and data concerning the size (for example,a vertical width and a lateral width calculated or measured in advance)of the visual confirmation screen S, as obtained from data of theoriginal image M corresponding to the visual confirmation screen S, canbe used as the reference data.

When a user performs an operation on the visual confirmation screen Susing a finger, the input control unit 76 recognizes (detects, analyzes)the contents of the inputted instruction (command) corresponding to theoperation performed by the finger based on: (i) the data concerning thecontents of the operation performed by the finger as obtained by adetermination made by the operation determining unit 73, (ii) thepositional data of the finger generated by the positional datagenerating unit 74, and (iii) the reference data concerning the visualconfirmation screen S stored in the storage unit 90; then, the inputcontrol unit 76 performs a control (process or operation) so that theoriginal screen M is displayed on the display apparatus 20 in accordancewith the recognized contents of the inputted instruction (command). Forexample, when the visual confirmation screen S is the character inputscreen 200 shown in FIG. 4, the input control unit 76 can recognize therange in which the character input screen 200 viewed by the user existswithin the imaging range of the imaging apparatus 30 based on thereference data concerning the visual confirmation screen S. In thiscase, since the input control unit 76 is aware of the configuration ofthe character input screen 200 in advance, the input control unit 76 canalso recognize the range (span) of the keyboard image 210, the region(area) of each character key image, and the like on the character inputscreen 200. Therefore, for example, when the user uses a finger toperform a touch operation on a character key image in the keyboard image210, the input control unit 76 can specify (identify) the operated(“touched”) character key by checking which character key image regionin the keyboard image 210, to which the position of the finger asobtained from the positional data of the finger corresponds.

Moreover, when the user uses a finger to perform an operation on thevisual confirmation screen S and the input control unit 76 recognizesthe contents of the inputted instruction (command) corresponding to theoperation performed by the finger, the input control unit 76 may specify(identify) the position on the visual confirmation screen S operated(“touched”) by the finger by first generating a reference screen, whichcorresponds to the visual confirmation screen S, on a virtual plane thatcorresponds to the imaging range of the imaging apparatus 30 using(based on) the reference data concerning the visual confirmation screenS stored in the storage unit 90. Then, the input control unit 76 maycheck which position of the reference screen, to which the positionaldata of the finger generated by the positional data generating unit 74corresponds.

The communicating unit 80 communicates (transmits/receives) informationwith the outside. The storage unit 90 stores various programs, data, andthe like. For example, programs stored in the storage unit 90 mayinclude: (i) a reference data setting program for performing a referencedata setting process (discussed below), (ii) a character input programfor performing, when the visual confirmation screen S is the characterinput screen 200, a character input process based on an operationperformed on the character input screen 200, and/or (iii) a screendisplay program for performing a screen display process includingenlargement/reduction and switching of the original image Mcorresponding to the visual confirmation screen S based on a manual(virtual) operation performed on the visual confirmation screen S. Inaddition, for example, data stored in the storage unit 90 may includeimage data of various types of original screens M, data concerning eachoriginal screen M (specifically, data indicating sizes, shapes,contents, configurations, and the like of the various types of originalscreens M), and various types of image data used when creating anoriginal screen for setting the reference data, which will be describedbelow. Furthermore, the storage unit 90 is also used as, or contains, aworking (volatile) memory (e.g., RAM).

With the computer display device 1 according to the first embodiment,when a user uses a finger to perform an operation on the visualconfirmation screen S, the input control unit 76 recognizes the contentsof the inputted instruction (command) corresponding to the operationperformed by the finger based on (i) the data concerning the contents ofthe operation performed by the finger as obtained by a determinationmade by the operation determining unit 73, (ii) the positional data ofthe finger generated by the positional data generating unit 74, and(iii) the reference data concerning the visual confirmation screen Sstored in the storage unit 90; the input control unit 76 then controls(selects or configures) the original screen M to be displayed on thedisplay apparatus 20 in accordance with the recognized contents of theinputted instruction. Therefore, by performing, on the visualconfirmation screen S that is viewed by the user, an operation similarto a manual operation performed on a screen displayed on a conventionaltouch panel (screen), the user can input an instruction (command)corresponding to the manual operation. In actual practice, when the userperforms a touch operation on the visual confirmation screen S using afinger, the input control unit 76 can recognize the instruction(command) corresponding to the touch operation in a manner similar totechniques used to recognize an instruction (command) when the visualconfirmation screen S is displayed on a conventional touch panel(screen). For example, when the user uses a finger to perform adouble-tapping operation on the visual confirmation screen S, the inputcontrol unit 76 recognizes an instruction (command) to enlarge or reducethe original screen M corresponding to the visual confirmation screen S.Similarly, when the user uses a finger to perform a long-tappingoperation on the visual confirmation screen S, the input control unit 76recognizes an instruction (command) to display a screen of option menusas the original screen M. Moreover, when the user uses a finger toperform a dragging operation on the visual confirmation screen S, theinput control unit 76 recognizes an instruction (command) to scroll anddisplay the original screen M. In addition, when the user uses a fingerto perform a touch operation on a character key image on the characterinput screen 200, the input control unit 76 recognizes an instruction(command) corresponding to the operation or, in other words, an inputinstruction of the character key and performs a process that displaysthe character corresponding to the inputted instruction on the originalscreen M in a manner similar to a character input screen 200 displayedon a touch panel.

Moreover, in the first embodiment, since the user uses a finger toperform a touch operation on the visual confirmation screen S, whichappears to be floating in midair, a touch operation can also beperformed in a mode (manner) that cannot be employed when performing atouch operation on a screen displayed on a conventional touch panel.FIGS. 6A to 7B are diagrams that explains additional modes (types) oftouch operations that can be performed on the visual confirmation screenS. That is, the user may normally or conventionally perform a touchoperation using one finger from on the front side of the visualconfirmation screen S as shown in FIG. 6A. However, in the presentembodiment, the user can also perform a touch operation using one fingerfrom the rear side of the visual confirmation screen S as shown in FIG.6B. In addition, the user can also perform a touch operation using aplurality of fingers from the front side of the visual confirmationscreen S as shown in FIG. 7A or can perform a touch operation using aplurality of fingers from the rear side of the visual confirmationscreen S as shown in FIG. 7B.

Next, an exemplary, non-limiting, process for setting reference data inthe computer display device 1 according to the first embodiment will bedescribed using FIG. 8, which is a flow chart that explains arepresentative, non-limiting procedure for a reference data settingprocess.

The user may, e.g., audibly (orally) issue an instruction (voicecommand) to set the reference data via the microphone unit 50 or may,e.g., issue an instruction to set the reference data by performing atouch operation using (by touching) the touchpad unit 40. Upon receivingsuch an instruction, the control unit 70 reads (accesses) the referencedata setting program from the storage unit 90 and performs (executes)the reference data setting process (program) in accordance with theprocessing flow (steps) shown in FIG. 8.

First, by adding an image of, for example, a circle at one or aplurality of prescribed positions on the original screen M currentlydisplayed on the display apparatus 20, the display control unit 71creates (generates) a new original screen M (original screen for settingthe reference data) and displays the new original screen M on thedisplay apparatus 20 (S1). For example, the circle image may be a markthat prompts the user to perform an operation using a finger at theposition of the circle (i.e. the user should virtually touch thecircle). FIGS. 9A-9C are diagrams showing an example of the originalscreen M that may be displayed during the process of setting thereference data. This example represents an embodiment in which theoriginal screen M is the character input screen 200. Initially, thenormal character input screen 200 shown in FIG. 9A is displayed on thedisplay apparatus 20. However, after the process of step S1 has beenexecuted, a modified character input screen 201 (original screen forsetting the reference data) shown in FIG. 9B is displayed on the displayapparatus 20. On the character input screen 201 shown in FIG. 9B, imagesrepresenting circles and numerals are added to (superimposed at) thepositions of the four corners of the character input screen 201. Whenthe character input screen 201 shown in FIG. 9B is displayed on thedisplay apparatus 20, the user can view the visual confirmation screen Scorresponding to the character input screen 201 (visual confirmationscreen for setting the reference data) or, in other words, a screenhaving the same contents as the character input screen 201 shown in FIG.9B. It is noted that, although images of circles are displayed at thefour corners of the character input screen 201 in FIG. 9B, images ofcircles may instead be displayed, e.g., at the four corners of thekeyboard image 210 on the character input screen 201 as shown in FIG.9C. The location of the “marks” is not particularly limited, as long asa predetermined spacing of the marks is stored in the storage unit 90 sothat the control unit 70 can appropriately generate the reference data.

After the process of step S1, the control unit 70 starts an imagingoperation performed by, or together with, the imaging apparatus 30 (S2).Upon viewing the character input screen 201 shown in FIG. 9B as thevisual confirmation screen S for setting the reference data, the userperforms a prescribed (predetermined) operation, such as a tappingoperation, using a finger on the respective numbered circles in anascending numerical order on the visual confirmation screen S to set thereference data. In this case, the user performs the prescribed operationin order to notify the control unit 70 of the positions at which theuser is performing the operation. One or more images of this operationby the user is (are) captured by the imaging apparatus 30. As thispoint, in the present embodiment, the imaging apparatus 30 captures animage of a subject (object) in focus. In addition, the image processingunit 32 performs prescribed image processing on the image data obtainedby the imaging operation; then the image data that was subjected to theimage processing is sent to the control unit 70 (S3).

Next, the image data extracting unit 72 determines whether or not thesubject (object) is a finger based on the image data obtained by theimaging operation performed by the imaging apparatus 30 and, if so,extracts image data in which the finger is present (S4). In thisembodiment, the imaging apparatus 30 sends image data, which wasobtained by capturing the subject (object) in focus, to the image dataextracting unit 72. Therefore, the image data extracting unit 72 willextract image data of the finger that is at a position separated(spaced) from the imaging apparatus 30 by an approximately constantdistance along the Z axis direction. Subsequently, the operationdetermining unit 73 determines whether or not the operation performed bythe finger is a prescribed operation (in this case, a tapping operation)based on the image data extracted by the image data extracting unit 72.The operation determining unit 73 performs such a determination processto determine whether or not tapping operations performed by the fingerwith respect to all four circles have been successfully (normally)recognized (S5). For example, the operation determining unit 73determines that tapping operations performed by the finger with respectto all four circles were not successfully (normally) recognized when adetermination that the operation performed by the finger is a tappingoperation that was done only once, twice, or three times within aprescribed (predetermined) period of time (set in advance) or when imagedata in which the finger is present is not sent from the image dataextracting unit 72 within a prescribed (predetermined) period of time(set in advance). When the operation determining unit 73 determines thattapping operations with the finger with respect to all four circles weresuccessfully (normally) recognized, the operation determining unit 73stores in the storage unit 90 data concerning the contents of theoperations performed by the finger and, at the same time, sends a signalinforming the display control unit 71 that the tapping operationsperformed by the finger have been successfully (normally) recognized.Subsequently, the display control unit 71 adds an image, e.g.,indicating a green lamp, to the original screen M signifying that thetapping operations performed by the finger have been successfully(normally) recognized and displays the original screen M on the displayapparatus 20 (S6). It is noted that, at this point, together with theimage indicating a green lamp or in place of the image indicating agreen lamp, the display control unit 71 may add images to the originalscreen M indicating one or more characters or graphics signifying thattapping operations performed by the finger have been successfully(normally) recognized. Alternatively, the control unit 70 may cause aspecific notification sound to be emitted from the speaker unit 60together with, or in place of, displaying an image signifying thattapping operations performed by the finger have been successfully(normally) recognized.

After the process of step S6, the positional data generating unit 74generates positional data (XY coordinates) of each finger (fingertip)operation in the imaging range of the imaging apparatus 30 based on theimage data for which the operation determining unit 73 has determined(confirmed) that the operation at each circle was a prescribed operation(S7). In addition, the reference data generating unit 75 stores the fourpieces of positional data generated in this manner in the storage unit90 as reference data concerning the visual confirmation screen Scurrently being displayed (S8). Since the reference data specifies(identifies) the position and the size of the visual confirmation screenS, by using the reference data the control unit 70 can recognize(determine) the range (span) of existence of the visual confirmationscreen S that is being viewed by the user within the imaging range ofthe imaging apparatus 30. After the process of step S8 is performed, thereference data setting process (program) is ended.

On the other hand, when the operation determining unit 73 determinesthat tapping operations performed by the finger with respect to all fourcircles were not successfully (normally) recognized in the process ofstep S5, the operation determining unit 73 sends a signal informing thedisplay control unit 71 that tapping operations performed by the fingerwere not successfully (normally) recognized. Subsequently, the displaycontrol unit 71 adds an image, e.g., indicating a red lamp, to theoriginal screen M signifying that tapping operations performed by thefinger have not been successfully (normally) recognized and displays theoriginal screen M on the display apparatus 20 (S9). Upon viewing theimage indicating the red lamp, the user must once again perform atapping operation using a finger on each circle on the visualconfirmation screen S to set the reference data. It is noted that, atthis point, together with the image indicating a red lamp or in place ofthe image indicating a red lamp, the display control unit 71 may addimages to the original screen M indicating one or more characters orgraphics signifying that tapping operations performed by the finger havenot been successfully (normally) recognized. Alternatively, the controlunit 70 may cause a specific notification sound to be emitted from thespeaker unit 60 together with, or in place of, displaying an imagesignifying that tapping operations performed by the finger have not beensuccessfully (normally) recognized.

After the process of step S9, the control unit 70 determines whether ornot the process of current step S5 is a first time process (S10). Whenthe process of current step S5 is a first time process (i.e. it is thefirst time that it has been performed after initiation of the referencedata setting process), the process returns to step S2. Furthermore, whenthe process of current step S5 is not a first time process, the controlunit 70 determines whether or not the process of current step S5 is asecond time process (S11). When the process of current step S5 is asecond time process (i.e. it is the second time that it has beenperformed after initiation of the reference data setting process), theprocess is again returned to step S2. On the other hand, when theprocess of current step S5 is not a first time process or a second timeprocess (i.e. it is the third time that the process has been performedunsuccessfully), the reference data setting process is ended. In otherwords, when the red lamp is displayed on the visual confirmation screenS, the user is given two further opportunities to successfully performthe necessary (tapping) operations using a finger. When an operationusing a finger is not successfully (normally) recognized even then, thecurrent reference data setting process may be terminated and then a newreference data setting process may be executed (initiated) once again.

In the present embodiment the operation determining unit 73 determineswhether or not an operation using a finger with respect to each circleis a tapping operation and determines whether or not tapping operationsusing a finger with respect to all four circles have been successfully(normally) recognized in step S5. However, the operation determiningunit 73 may instead determine every time the finger performs anoperation with respect to each circle whether or not the operation is atapping operation and also may determine whether or not the tappingoperation has been successfully (normally) recognized. In this case,every time the operation determining unit 73 determines that a tappingoperation performed using a finger with respect to each circle has beensuccessfully (normally) recognized, the display control unit 71 maypreferably display on the original screen M an image signifying that thetapping operation performed by the finger with respect to the circle wassuccessfully (normally) recognized; furthermore, every time theoperation determining unit 73 determines that a tapping operationperformed by a finger with respect to each circle has not beensuccessfully (normally) recognized, the display control unit 71 maypreferably display on the original screen M an image signifying that atapping operation performed by a finger with respect to the circle hasnot been successfully (normally) recognized. Specifically,representative, non-limiting examples of an image signifying that atapping operation performed by a finger with respect to each circle hasbeen successfully (normally) recognized may include, e.g., an image thatinversely displays the circle and an image displaying the color of thecircle in green; furthermore, an example of an image signifying that atapping operation performed by a finger with respect to each circle hasnot been successfully (normally) recognized may be an image displayingthe color of the circle in red. In addition or in the alternative, thecontrol unit 70 may cause a specific notification sound to be emittedfrom the speaker unit 60 together with, or in place of, an imagesignifying that a tapping operation performed by a finger with respectto each circle has been successfully (normally) recognized or an imagesignifying that a tapping operation performed by a finger with respectto the circle has not been successfully (normally) recognized.

Furthermore, in the present embodiment the user performs a prescribedoperation using a finger with respect to four prescribed locations onthe visual confirmation screen S to acquire four pieces of positionaldata. However, the user may instead perform a prescribed operation usinga finger with respect to one, two, or three prescribed locations on thevisual confirmation screen S to acquire positional data of the one, two,or three locations. But, in this case, data concerning the size of thevisual confirmation screen S must be calculated in advance bycomputation from data of the original screen M corresponding to thevisual confirmation screen S and stored in the storage unit 90.Furthermore, the respective pieces of positional data and dataconcerning the size of the visual confirmation screen S constitute thereference data.

Next, a character input process in the computer display device 1according to the first embodiment will be described using FIG. 10, whichis a flow chart that explains a procedure of a representative,non-limiting process for inputting characters. In the followingdescription, it is assumed that the character input screen 200 isdisplayed in advance on the display apparatus 20 as the original screenM.

To begin the process, the user first issues a voice instruction(command) to perform a character input via the microphone unit 50 orissues (inputs) a manual instruction to perform character input usingthe touchpad unit 40, or the like. Upon receiving the instruction, thecontrol unit 70 reads the character input program for the characterinputting process from the storage unit 90 and executes (initiates) thecharacter inputting process in accordance with the processing flow shownin FIG. 10. It is noted that the character input process may beconfigured to be automatically executed (initiated) when the characterinput screen 200 is displayed on the display apparatus 20 as theoriginal screen M.

Then, the control unit 70 determines whether or not reference dataconcerning the visual confirmation screen S corresponding to theoriginal screen M currently being displayed on the display apparatus 20is stored in the storage unit 90 (S21). When reference data concerningthe visual confirmation screen S is not stored in the storage unit 90,the control unit 70 reads the reference data setting program from thestorage unit 90 and executes the reference data setting process inaccordance with the processing flow shown in FIG. 8 (S22). Subsequently,the process is returned to step S21. It is noted that, although thepresent embodiment executes the reference data setting process whenreference data concerning the visual confirmation screen S is not storedin the storage unit 90, even if reference data concerning the visualconfirmation screen S is stored in the storage unit 90, the referencedata setting process may be executed to once again generate thereference data when an instruction is received from the user.

When it is determined in step S21 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 (S23). The user performs a prescribedoperation such as a tapping operation using a finger on the keyboardimage 210 of the character input screen 200 that is the visualconfirmation screen S. In this case, the user performs the prescribedoperation in order to notify the control unit 70 of the position atwhich the user is performing the operation. One or more images of theoperation by the user is (are) captured by the imaging apparatus 30 andthe obtained image data is sent to the image processing unit 32. Inaddition, the image processing unit 32 performs prescribed imageprocessing on the image data and the image data, which was subjected tothe image processing, is sent to the control unit 70 (S24).

Next, the image data extracting unit 72 determines whether or not thesubject (object) is a finger based on the image data obtained by imagingperformed by the imaging apparatus 30 and, if so, extracts image data inwhich the finger is present (S25). In other words, the image dataextracting unit 72 extracts image data of the finger that is at aposition separated (spaced) from the imaging apparatus 30 by anapproximately constant distance along the Z axis direction.Subsequently, the operation determining unit 73 determines whether ornot the operation performed by the finger is a prescribed operation (inthis case, a tapping operation) based on the image data extracted by theimage data extracting unit 72. This determination is made within aprescribed (predetermined) period of time (set in advance). In addition,when the operation performed by the finger is a tapping operation, theoperation determining unit 73 determines that a character inputoperation has been successfully (normally) recognized, and when theoperation performed by the finger is not a tapping operation, theoperation determining unit 73 determines that a character inputoperation has not been successfully (normally) recognized (S26). When itis determined that a character input operation has been successfully(normally) recognized, the operation determining unit 73 stores dataconcerning the contents of the operation performed by the finger in thestorage unit 90 and, at the same time, sends a signal informing thedisplay control unit 71 that a character input operation has beensuccessfully (normally) recognized. When the display control unit 71receives the signal, the display control unit 71 adds an image, e.g.,indicating a green lamp, to the original screen M signifying that theoperation for character input has been successfully (normally)recognized and displays the original screen M on the display apparatus20 (S28). It is noted that, together with the image indicating a greenlamp or in place of the image indicating a green lamp, the displaycontrol unit 71 may (also) add one or more images to the original screenM indicating one or more characters or graphics signifying that thecharacter input operation has been successfully (normally) recognized.Alternatively, as was described above, the control unit 70 may cause aspecific notification sound to be emitted from the speaker unit 60together with, or in place of, displaying an image signifying that thecharacter input operation has been successfully (normally) recognized.

On the other hand, when the operation determining unit 73 determines instep S26 that a character input operation has not been successfully(normally) recognized within a prescribed (predetermined) period of time(set in advance), the operation determining unit 73 sends a signal tothat effect to the display control unit 71. At this point, for example,if image data in which a finger is present is not sent from the imagedata extracting unit 72 within the prescribed (predetermined) period oftime (set in advance), the operation determining unit 73 determines thata character input operation has not been successfully (normally)recognized. When the display control unit 71 receives the signal, thedisplay control unit 71 adds an image, e.g., indicating a red lamp, tothe original screen M signifying that a character input operation hasnot been successfully (normally) recognized and displays the originalscreen M on the display apparatus 20 (S27). Subsequently, the processreturns to step S32. It is noted that, at this point, together with theimage indicating a red lamp or in place of the image indicating a redlamp, the display control unit 71 may (also) add one or more imagesindicating characters or graphics to the original screen M signifyingthat a character input operation has not been successfully (normally)recognized. Alternatively, as was described above, the control unit 70may cause a specific notification sound to be emitted from the speakerunit 60 together with, or in place of, displaying an image signifyingthat character input the operation has not been successfully (normally)recognized.

After the process of step S28, the positional data generating unit 74generates positional data of the finger (fingertip) in the imaging rangeof the imaging apparatus 30 based on image data for which the operationdetermining unit 73 has determined the operation performed by the fingeris a tapping operation (S29). Positional data of a finger generated inthis manner is stored in the storage unit 90.

Next, the input control unit 76 recognizes the contents of an inputtedinstruction corresponding to the operation performed by the finger basedon: (i) the data concerning the contents of the operation performed bythe finger as obtained by a determination by the operation determiningunit 73, (ii) the positional data of the finger generated by thepositional data generating unit 74, and (iii) the reference dataconcerning the visual confirmation screen S stored in the storage unit90 (S30). For example, when the user performs a tapping operation usinga finger with respect to a character key image in the keyboard image210, by checking which character key image region in the keyboard image210 to which the position of the finger obtained from positional data ofthe finger corresponds, the input control unit 76 can specify (identify)the character key on which the current tapping operation has beenperformed and recognize that input of the specified character key hasbeen instructed (is intended by the user). Subsequently, the inputcontrol unit 76 sends a signal to the display control unit 71 concerningthe recognized contents of the inputted instruction and the displaycontrol unit 71 displays the original screen M corresponding to thecontents of the inputted instruction on the display apparatus 20 (S31).

After the process of step S31 or step S27, the control unit 70determines whether or not an instruction to end (terminate) thecharacter input procedure has been received from the user (S32). When aninstruction to end character input procedure has been received, thecharacter input process is ended. On the other hand, when an instructionto end character input procedure has not been received, the processreturns to step S23 and the character input process is continued. It isnoted that the user may issue the instruction (command) to end characterinput procedure by voice, by touching the touchpad unit 40, by virtuallytouching a predetermined location on the keyboard image 210, or thelike.

Next, a screen display process in the computer display device 1according to the first embodiment will be described with reference toFIG. 11, which is a flow chart that explains representative,non-limiting steps of the screen display process.

To begin the process, the user first issues a voice instruction(command) to perform an operation for screen display via the microphoneunit 50 or issues (inputs) a manual instruction to perform an operationfor screen display, e.g., using the touchpad unit 40, the visualconfirmation screen S or the like. Upon receiving the instruction, thecontrol unit 70 reads the screen display program from the storage unit90 and executes (initiates) a screen displaying process in accordancewith a processing flow shown in FIG. 11. It is noted that the screendisplaying process (program) may be configured to be automaticallyexecuted when the original screen M is displayed on the displayapparatus 20.

Then, the control unit 70 determines whether or not reference dataconcerning the visual confirmation screen S corresponding to theoriginal screen M currently being displayed on the display apparatus 20is stored in the storage unit 90 (S41). When reference data concerningthe visual confirmation screen S is not stored in the storage unit 90,the control unit 70 reads the reference data setting program from thestorage unit 90 and executes the reference data setting process inaccordance with the processing flow shown in FIG. 8 (S42). Subsequently,the process returns to step S41. It is noted that, although the presentembodiment executes the reference data setting process when referencedata concerning the visual confirmation screen S is not stored in thestorage unit 90, even if reference data concerning the visualconfirmation screen S is stored in the storage unit 90, the referencedata setting process may be executed to once again generate referencedata when an instruction is received from the user.

When it is determined in step S41 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 (S43). The user performs a desiredoperation using a finger on the visual confirmation screen S. One ormore images of the operation performed by the user (the user's finger)is (are) captured by the imaging apparatus 30 and the obtained imagedata is sent to the image processing unit 32. In addition, the imageprocessing unit 32 performs prescribed image processing on the imagedata and the image data, which has been subjected to the imageprocessing, is sent to the control unit 70 (S44).

Next, the image data extracting unit 72 determines whether or not thesubject (object) is a finger based on the image data obtained by theimaging performed by the imaging apparatus 30 and, if so, extracts imagedata in which the finger is present (S45). In other words, the imagedata extracting unit 72 extracts image data of the finger that is at aposition separated (spaced) from the imaging apparatus 30 by anapproximately constant distance along the Z axis direction.Subsequently, the operation determining unit 73 determines the contentsof the operation performed by the finger based on the image dataextracted by the image data extracting unit 72. In addition, theoperation determining unit 73 determines whether or not the operationperformed by the finger was successfully (normally) recognized (S46).When it is determined that an operation performed by the finger wassuccessfully (normally) recognized, the operation determining unit 73stores in the storage unit 90 data concerning the contents of theoperation performed by the finger and, at the same time, sends a signalinforming the display control unit 71 that the operation performed bythe finger has been successfully (normally) recognized. When the displaycontrol unit 71 receives the signal, the display control unit 71 adds animage, e.g., indicating a green lamp, to the original screen Msignifying that the operation performed by the finger has beensuccessfully (normally) recognized and displays the original screen M onthe display apparatus 20 (S48). It is noted that, together with theimage indicating a green lamp or in place of the image indicating agreen lamp, the display control unit 71 may (also) add images to theoriginal screen M indicating one or more characters or graphics thatsignify the operation performed by the finger has been successfully(normally) recognized. Alternatively, as was described above, thecontrol unit 70 may cause a specific notification sound to be emittedfrom the speaker unit 60 together with, or in place of, displaying animage signifying that the operation performed by the finger has beensuccessfully (normally) recognized.

On the other hand, when the operation determining unit 73 determines instep S46 that an operation performed by the finger has not beensuccessfully (normally) recognized, the operation determining unit 73sends a signal to that effect to the display control unit 71. At thispoint, for example, if image data in which a finger is present is notsent from the image data extracting unit 72 within the prescribed(predetermined) period of time (set in advance), the operationdetermining unit 73 determines that a tapping operation has not beensuccessfully (normally) recognized. When the display control unit 71receives the signal, the display control unit 71 adds an image, e.g.,indicating a red lamp, to the original screen M signifying that anoperation performed by the finger has not been successfully (normally)recognized and displays the original screen M on the display apparatus20 (S47). Subsequently, the process returns to step S52. It is notedthat, at this point, together with the image indicating a red lamp or inplace of the image indicating a red lamp, the display control unit 71may (also) add images to the original screen M indicating one or morecharacters or graphics that signify the operation performed by thefinger has not been successfully (normally) recognized. Alternatively,as was described above, the control unit 70 may cause a specificnotification sound to be emitted from the speaker unit 60 together with,or in place of, displaying an image signifying that the operationperformed by the finger has not been successfully (normally) recognized.

After the process of step S48, the positional data generating unit 74generates positional data of the finger (fingertip) in the imaging rangeof the imaging apparatus 30 based on image data representing thecontents of the operation performed by the finger as determined by theoperation determining unit 73 (S49). Positional data of the fingergenerated in this manner is stored in the storage unit 90.

Next, the input control unit 76 recognizes the contents of aninstruction corresponding to the operation performed by the finger basedon: (i) the data concerning the contents of the operation performed bythe finger as obtained by a determination by the operation determiningunit 73, (ii) the positional data of the finger generated by thepositional data generating unit 74, and (iii) the reference dataconcerning the visual confirmation screen S stored in the storage unit90 (S50). For example, when the user performs a double-tapping operationon the visual confirmation screen S using a finger, the input controlunit 76 specifies (identifies) that the current operation is adouble-tapping operation and recognizes (determines) that an instructionto enlarge (or reduce) the original screen M has been received.Subsequently, the input control unit 76 sends a signal concerning therecognized contents of the instruction to the display control unit 71and the display control unit 71 displays the original screen M inaccordance with the contents of the instruction on the display apparatus20 (S51), i.e. the size of the original screen M may be enlarged orreduced.

After the process of step S51 or step S47, the control unit 70determines whether or not an instruction (command) to end (terminate)the screen display operation has been received from the user (S52). Whenan instruction to end the screen display operation has been received,the screen display process is ended (terminated). On the other hand,when an instruction to end the screen display operation has not beenreceived, the process returns to step S43 and the screen display processis continued. It is noted that the user may issue (input) theinstruction command to end the screen display operation by voice, bytouch using the touchpad unit 40, or the virtual confirmation screen, orthe like.

With the computer display device according to the first embodiment, whena user performs an operation on the visual confirmation screen using afinger, the input control unit recognizes (identifies or determines) thecontents of the inputted instruction corresponding to the operationperformed by the finger based on: (i) the data concerning the contentsof the operation performed by the finger as obtained by a determinationmade by the operation determining unit, (ii) the positional data of thefinger generated by the positional data generating unit, and (iii) thereference data concerning the visual confirmation screen and stored inthe storage unit; the input control unit then controls (causes) theoriginal screen to be displayed on the display apparatus in accordancewith the recognized contents of the inputted instruction. Therefore, byperforming, on a visual confirmation screen that appears as though it isfloating in midair, an operation similar to a manual operation(manipulation) performed on a screen displayed on a conventional touchpanel, the user can input an instruction (command) corresponding to themanual operation. Consequently, by using the computer display deviceaccording to the first embodiment, the user can easily and accuratelyinput characters and perform various screen operations, includingenlargement and reduction, by operating a visual confirmation screen ina manner similar to a conventional smartphone screen or a tablet screen.

Modification of the First Embodiment

Next, a computer display device according to a modification of the firstembodiment of the present teachings will be described with reference toFIGS. 21A to 22B. Note that, in the present and all subsequentmodifications and alternate embodiments, components sharing the samefunctions as those in the first embodiment described above will bedenoted by the same reference numerals and detailed descriptions thereofwill be omitted.

As shown in FIGS. 21A to 22B, the computer display device 1 d accordingto the present modification includes many of the same components as weredescribed above in the first embodiment.

The present computer display device 1 d only differs from the computerdisplay device 1 according to the first embodiment in the configurationof the display apparatus 20 d. Specifically, the display apparatus 20 dis comprised of a small projector 21 that includes a liquid crystalpanel (display device) or the like, an optical system 22, and a halfmirror 23. The half mirror 23 is embedded in the right-side lens section11 b of the eyeglass main body 10 d as shown in FIGS. 21A to 22B. Anoriginal screen M displayed on the liquid crystal panel is projected onthe half mirror 23 via the optical system as shown in FIGS. 21A to 22B.

Operational advantages of the computer display device 1 d according tothe present modification are similar to those of the first embodimentdescribed earlier. Therefore, by using the presently-modified computerdisplay device 1 d, various screen operations such as character inputand enlargement and reduction can be easily and accurately performed ona visual confirmation screen displayed in the field of vision.

Second Embodiment

Next, a computer display device according to a second embodiment of thepresent teachings will be described with reference to FIG. 12.

As shown in FIG. 12, the computer display device 1 a according to thesecond embodiment includes the same components as were described abovein the first embodiment.

The present computer display device 1 a differs from the computerdisplay device according to the first embodiment in that the cameracontrol unit 33 a includes an autofocus control unit 331 and that theimage data extracting unit 72 a extracts image data, in which thesubject (object) is a finger and the finger is at a position separated(spaced) by an approximately constant distance from the imagingapparatus 30 a in the Z axis direction, from image data sent from theimaging apparatus 30 a.

The autofocus control unit 311 is configured or adapted to control thecamera unit 31 so that a subject (object) at a prescribed positionwithin the imaging range is automatically brought into focus. In thesecond embodiment, the imaging apparatus 30 a has a large number offocal points so that any position within the imaging range can beautomatically brought into focus. Therefore, when the user is performingan operation on the visual confirmation screen S using a finger, theimaging apparatus 30 a is capable of automatically bringing the fingerinto focus and capturing images in a state in which the finger is infocus. In addition, when capturing an image of the subject (object) thathas been automatically brought into focus, the autofocus control unit331 calculates distance data to the captured subject (object). Thecalculated distance data is then associated with the image data. Inaddition, the image data obtained by the imaging operation performed bythe imaging apparatus 30 a and the distance data associated with theimage data are sent to the control unit 70 a. It is noted that, theautofocus system may be, e.g., either: (i) an active system, in whichthe subject is irradiated with infrared rays, ultrasonic waves, or thelike and the distance is determined based on the period of time untilreflected waves are detected or based on an irradiation angle, or (ii) apassive system, such as a phase difference detection system or acontrast detection system, that measures the distance using an imagecaptured by the lens of the camera unit 31.

In addition, when the user performs an operation on the visualconfirmation screen S using a finger and the imaging apparatus 30 acaptures an image of a subject (finger) in focus, the image dataextracting unit 72 a extracts image data containing the finger (as thesubject/object) and the finger is at a position separated (spaced) by anapproximately constant distance from the imaging apparatus 30 a in the Zaxis direction. This process is performed by (i) determining whether ornot the subject is a finger based on image data obtained by the imagingand, at the same time, (ii) determining whether or not the subject(finger) is separated (spaced) by an approximately constant distance(set in advance) from the imaging apparatus 30 a in the Z axis directionbased on the distance data associated with the image data that wasobtained by the above-described imaging operation. Known image (object)recognition methods and/or known image (object) recognition software maybe used to determine whether or not the subject is a finger in a mannersimilar to the first embodiment. In addition, when determining whetheror not the subject is separated (spaced) by an approximately constantdistance from the imaging apparatus 30 a in the Z axis direction, theapproximately constant distance refers to the distance in the Z axisdirection from the imaging apparatus 30 a to the position of the visualconfirmation screen S as recognized by the user. For example, when theuser recognizes that the visual confirmation screen S is at a positionseparated (spaced) by approximately 40 cm from the imaging apparatus 30a, the above-mentioned approximately constant distance is set to adistance within a range of approximately 40 cm±5 cm from the imagingapparatus 30 a. As described above, the image data extracting unit 72 acan eliminate image data of the finger performing an operation at aposition that is significantly in front of or behind the position wherethe visual confirmation screen S is being displayed and can therebyextract image data of the finger only when it is performing anappropriate operation on the visual confirmation screen S. It is notedthat, the operation determining unit 73, the positional data generatingunit 74, and the reference data generating unit 75 perform additionalprocesses based on the image data extracted by the image data extractingunit 72 a.

In a manner similar to the first embodiment, when the user performs anoperation using a finger at one or a plurality of prescribed positionson the visual confirmation screen S, the reference data generating unit75 generates data concerning the visual confirmation screen S asreference data using the positional data of the finger generated by thepositional data generating unit 74 based on image data for which theoperation determining unit 73 has determined an operation at eachprescribed position is a prescribed operation. For example, when theuser performs an operation using a finger with respect to the fourcorners of the outer frame of the visual confirmation screen S, thepositional data of the finger at each position of the four corners canbe used as reference data. As was described above, in the secondembodiment as well, since image data extracted by the image dataextracting unit 72 a represents a captured image of a finger at aposition separated (spaced) by an approximately constant position alongthe Z axis direction from the imaging apparatus 30 a, it isconsidered/deemed that the positional data of the finger at eachposition of the four corners represents positional information of thefinger on a plane parallel to an XY plane (approximately parallel to theuser's body) at a position separated (spaced) from the imaging apparatus30 a by an approximately constant distance along the Z axis direction.

Next, a process for setting reference data in the computer displaydevice 1 a according to the second embodiment will be described usingthe flow chart according to the first embodiment shown in FIG. 8.

The reference data setting processes according to the first and secondembodiments differ in the processes (steps S2 and S3) performed by theimaging apparatus 30 a and the process (step S4) of extracting the imagedata that is performed by the image data extracting unit 72 a.Therefore, only the different processes of the reference data settingprocess according to the second embodiment will be described withreference to the flow chart shown in FIG. 8.

After the process of step Si, the control unit 70 a starts an imagingoperation performed by, or together with, the imaging apparatus 30 a(S2). Upon viewing the character input screen 201 shown in FIG. 9B asthe visual confirmation screen S for setting the reference data, theuser performs a prescribed operation such as a tapping operation using afinger on the respective numbered circles in an ascending numericalorder on the visual confirmation screen S to set the reference data. Oneor more images of the operation by the user is (are) captured by theimaging apparatus 30 a. At this point, the autofocus control unit 331controls the camera unit 31 so that a subject (finger) within theimaging range is automatically brought into focus and the imagingapparatus 30 a captures one or more images of the subject (finger) infocus. In addition, when capturing an image of the subject (finger) thathas been automatically brought into focus, the autofocus control unit331 calculates data concerning the distance to the captured subject andthen associates the calculated distance data with the image data. Theimage data obtained by this imaging operation is sent to the imageprocessing unit 32 and the image processing unit 32 performs prescribedimage processing on the image data. Subsequently, the image data, whichhas been subjected to the image processing, and the distance dataassociated with the image data are sent to the control unit 70 a (S3).

In step S4, the image data extracting unit 72 a first determines whetheror not the subject is a finger based on the image data obtained byimaging performed by the imaging apparatus 30 a to extract image data inwhich the finger is present. Subsequently, by determining whether or notthe subject is separated (spaced) by an approximately constant distancefrom the imaging apparatus 30 a in the Z axis direction based on thedistance data associated with the extracted image data in which thefinger is present, the image data extracting unit 72 a extracts imagedata whose subject is the finger and the finger is at a positionseparated (spaced) by an approximately constant distance from theimaging apparatus 30 a in the Z axis direction. Note that each of theprocesses of the following steps (S5-S11) in the reference data settingprocess according to the second embodiment may be the same or similar tothose of the first embodiment as shown in FIG. 8.

Next, a character input process in the computer display device 1 aaccording to the second embodiment will be described with respect theflow chart according to the first embodiment shown in FIG. 10.

The character input processes according to the first and secondembodiments differ in the processes (steps S23 and S24) performed by theimaging apparatus 30 a and the process (step S25) of extracting theimage data that is performed by the image data extracting unit.Therefore, only the different processes of the character input processaccording to the second embodiment will be described with reference tothe flow chart shown in FIG. 10.

When it is determined in step S21 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 a starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 a (S23). The user performs a prescribedoperation such as a tapping operation using a finger on the keyboardimage 210 of the character input screen 200 that is the visualconfirmation screen S. One or more images of the operation performed bythe user (the user's finger) is (are) captured by the imaging apparatus30 a. At this point, the autofocus control unit 331 controls the cameraunit 31 so that a subject (finger) within the imaging range isautomatically brought into focus and the imaging apparatus 30 a capturesone or more images of the subject (finger) in focus. In addition, whencapturing an image of the subject (finger) that has been automaticallybrought into focus, the autofocus control unit 331 calculates dataconcerning the distance to the captured subject and then associates thecalculated distance data with the image data. The image data obtained bythis imaging operation is then sent to the image processing unit 32 andthe image processing unit 32 performs prescribed image processing on theimage data. Subsequently, the image data, which has been subjected tothe image processing, and the distance data associated with the imagedata are sent to the control unit 70 a (S24).

In step S25, the image data extracting unit 72 a first determineswhether or not the subject is a finger based on the image data obtainedby the imaging operation performed by the imaging apparatus 30 a toextract image data in which the finger is present. Subsequently, bydetermining whether or not the subject is separated (spaced) by anapproximately constant distance from the imaging apparatus 30 a in the Zaxis direction based on the distance data associated with the extractedimage data in which the finger is present, the image data extractingunit 72 a extracts image data whose subject (object) is the finger andthe finger is at a position separated (spaced) by an approximatelyconstant distance from the imaging apparatus 30 a in the Z axisdirection. Note that each of the following processes (S26-S32) in thecharacter input process according to the second embodiment are the sameor similar to those of the first embodiment as shown in FIG. 10.

Next, a screen display process in the computer display device 1 aaccording to the second embodiment will be described with reference tothe flow chart according to the first embodiment shown in FIG. 11.

The screen display processes according to the first and secondembodiments differ in the processes (steps S43 and S44) performed by theimaging apparatus 30 a and the process (step S45) of extracting theimage data that is performed by the image data extracting unit.Therefore, only the different processes of the screen display processaccording to the second embodiment will be described with reference tothe flow chart shown in FIG. 11.

When it is determined in step S41 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 a starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 a (S43). The user performs a desiredoperation using a finger on the visual confirmation screen S. One ormore images of the operation by the user (the user's finger) is (are)captured by the imaging apparatus 30. At this point, the autofocuscontrol unit 331 controls the camera unit 31 so that a subject (finger)within the imaging range is automatically brought into focus and theimaging apparatus 30 a captures one or more images of the subject(finger) in focus. In addition, when capturing an image of the subject(finger) that has been automatically brought into focus, the autofocuscontrol unit 331 calculates data concerning the distance to the capturedsubject and then associates the calculated distance data with the imagedata. The image data obtained by this imaging operation is then sent tothe image processing unit 32 and the image processing unit 32 performsprescribed image processing on the image data. Subsequently, the imagedata, which has been subjected to the image processing, and the distancedata associated with the image data are sent to the control unit 70 a(S24).

In step S45, the image data extracting unit 72 a first determineswhether or not the subject is a finger based on the image data obtainedby imaging performed by the imaging apparatus 30 a to extract image datain which the finger is present. Subsequently, by determining whether ornot the subject is separated (spaced) by an approximately constantdistance from the imaging apparatus 30 a in the Z axis direction basedon the distance data associated with the extracted image data in whichthe finger is present, the image data extracting unit 72 a extractsimage data whose subject is the finger and the finger is at a positionseparated (spaced) by an approximately constant distance from theimaging apparatus 30 a in the Z axis direction. Note that each of thefollowing processes (S46-S52) in the screen display process according tothe second embodiment may be the same or similar to those of the firstembodiment as shown in FIG. 11.

Operational advantages of the computer display device according to thesecond embodiment are the same or similar to those of the computerdisplay device according to the first embodiment. In particular, in thesecond embodiment, the imaging apparatus includes an autofocus controlunit capable of automatically bringing a subject (e.g., a finger) intofocus, and when an image of a subject (finger) that has beenautomatically brought into focus by the autofocus control unit iscaptured, distance data to the captured subject is calculated and thecalculated distance data is output together with image data obtained bythe imaging operation. Consequently, one or more images of a finger(fingertip), i.e. the subject/object, can be captured by bringing thesubject/object more accurately into focus. Therefore, the control unitcan more accurately generate the reference data, perform a characterinput process and the like, based on the image data obtained by theimaging operation and on the distance data.

It is noted that, in the computer display device according to the secondembodiment, the display apparatus 20 d according to the modification ofthe first embodiment may be used as the display apparatus. That is, asshown in FIGS. 21A to 22B, the display apparatus 20 d may include asmall projector 21, an optical system 22, and a half mirror 23 embeddedin a right-side lens section 11 b of the eyeglass main body 10 d.

First Modification of the Second Embodiment

Next, a computer display device according to a first modification of thesecond embodiment of the present teachings will be described withreference to FIG. 13.

As shown in FIG. 13, the computer display device 1 b according to thefirst modification of the second embodiment includes the same componentsas FIG. 12. However, the control unit 70 b of FIG. 13 further includes adeviation correcting unit 77 b.

In the present modification, it is noted that the plane corresponding tothe visual confirmation screen S, which is obtained based on thereference data concerning the visual confirmation screen S stored in thestorage unit 90, will be referred to as “reference screen K”.

For example, when inputting characters, the user may recognize that thescreen to be actually operated, touched, virtually manipulated, etc.(hereinafter, also referred to as “operation screen T”) is positioned infront of or behind reference screen K that is obtained based on thereference data; the user may then perform an operation on operationscreen T using a finger. In other words, a deviation may occur betweenthe operation screen T and reference screen K. When the deviation islarge, there is a possibility that the control unit 70 b may not be ableto accurately determine to which position on reference screen K theposition of a finger corresponds when the user performs an operationusing the finger. The deviation correcting unit 77 b executes a process(program) to convert positional data of the finger, which is obtained inaccordance with the manual (actual) operation performed by the user onoperation screen T using the finger, into positional data on referencescreen K. In this case, the positional data of the finger obtained by anoperation performed by the user on operation screen T using the fingeris positional data that has been generated by the positional datagenerating unit 74 b.

Next, a process for converting the positional data using the deviationcorrecting unit 77 b will be described in further detail. The conversionprocess is individually (separately) performed for the X coordinate andthe Y coordinate of the positional data. FIG. 14 shows a diagram thatwill be used to explain an exemplary process executed by the deviationcorrecting unit 77 b to convert the X coordinate of the positional data(on operation screen T) into the X coordinate of the positional data onreference screen K according to the first modification of the secondembodiment. FIG. 15 shows a diagram that will be used to explain anexemplary process executed by the deviation correcting unit 77 b toconvert the Y coordinate of the positional data (on operation screen T)into the Y coordinate of the positional data on reference screen Kaccording to the first modification of the second embodiment. FIGS. 14and 15 concern an embodiment in which the user recognizes that operationscreen T is positioned behind reference screen K.

In FIGS. 14 and 15, point Cc represents the center position of thecamera unit 31, point Mc represents the center position of the originalscreen M, and point Ec represents the center position of a pupil of theuser. Point pc represents the center position of reference screen K andpoint Pc represents the center position of operation screen T. In thiscase, point Pc, point pc, point Mc, and point Ec exist on the samestraight line. In addition, W denotes the distance between the centerposition of the camera unit 31 and the center position of originalscreen M in the X axis direction, H denotes the distance between thecenter position of the camera unit 31 and the center position oforiginal screen M in the Y axis direction, L denotes the distancebetween original screen M and reference screen K in the Z axisdirection, and a denotes the distance between the pupil of the user andoriginal screen M in the Z axis direction. The values of W, H, and a arestored in advance in the storage unit 90; the value of L is obtainedwhen generating the reference data (see above) and is stored in thestorage unit 90.

Assuming that the user has performed an operation using a finger atpoint P on operation screen T, the point where a straight lineconnecting point P and point Mc intersects reference screen K is denotedby p0 and the point where a straight line connecting point P and pointCc intersects reference screen K is denoted by p1. In the present(modified) embodiment, the positional data generating unit 74 b acquiresthe XY coordinates, when the actual position of the finger is projectedonto reference screen K, as the positional data of the finger.Therefore, the positional data generating unit 74 b calculates thepositional data of point p1 as the positional data of point P. Inaddition, the distance between point P and original screen M in the Zaxis direction or, in other words, the distance (Z) between operationscreen T and original screen M in the Z axis direction is obtained fromthe distance data associated with the image data used when generatingthe positional data of point P. Since point p0 is the point on referencescreen K that corresponds to point P on operation screen T, thedeviation correcting unit 77 b should obtain the positional data ofpoint p0 from the positional data of point p1. It is noted that,hereinafter, the position coordinates of point P are denoted as (X, Y),the position coordinates of point p0 are denoted as (x0, y0), theposition coordinates of point pc are denoted as (xc, yc), the positioncoordinates of point Pc are denoted as (Xc, Yc), and the positioncoordinates of point p1 are denoted as (x1, y1). Accordingly, sincepoint pc is the center position of reference screen K, the positioncoordinates (xc, yc) thereof are known and are stored in the storageunit 90. Furthermore, the point where the straight line connecting pointpc and point Cc intersects operation screen T is denoted as Pd, thedistance between point Pd and point P in the X axis direction is denotedas dX, and the distance between point Pd and point P in the Y axisdirection is denoted as dY.

An expression in which x0 is expressed using X can be obtained asfollows. First, in FIG. 14, focusing on triangle pc-Pd-Pc and trianglepc-Cc-Mc, from dX:W=(Z−L):L, it is possible to obtain:

dX=W×(Z−L)/L.

In addition, focusing on triangle Cc-Pd-P and triangle Cc-pc-p1, from{(X−Xc)+dX}:(x1−xc)=Z:L, it is possible to obtain:

X−Xc=(x1−xc)×Z/L−dX=(x1−xc)×Z/L−W×(Z−L)/L.

Furthermore, focusing on triangle Ec-Pc-P and triangle Ec-pc-p0, from(X−Xc):(x0−xc)=(Z+α):(L+α), it is possible to obtain:

x0−xc=(X−Xc)×(L+α)/(Z+α)={(x1−xc)×Z/L−W×(Z−L)/L}×(L+α)/(Z+α).

Therefore:

x0=(x0−xc)+xc={(x1−xc)×Z/L−W×(Z−L)/L}×(L+α)/(Z+α)+xc  (1)

Furthermore, by applying the same logic (mathematical derivations) toFIG. 15, an expression in which y0 is expressed using Y can be obtainedas:

y0=(y0−yc)+yc={(y1−yc)×Z/L−H×(Z−L)/L}×(L+α)/(Z+α)+yc  (2)

Both expressions (1) and (2) are also true when the user recognizes thatoperation screen T is positioned in front of reference screen K.

By substituting the value of the positional data (x1, y1) of point p1,which is generated by the positional data generating unit 74 b, and thevalue of the distance (Z) between point P and original screen M in the Zaxis direction into the above-noted expressions (1) and (2), thedeviation correcting unit 77 b can obtain the positional data (x0, y0)of point p0.

When a user performs an operation using a finger, the input control unit76 recognizes the contents of the inputted instruction corresponding tothe operation performed by the finger based on: (i) the data concerningthe contents of the operation performed by the finger as obtained by adetermination made by the operation determining unit 73, (ii) thepositional data (x0, y0) of the finger as obtained by the deviationcorrecting unit 77 b, and (iii) the reference data concerning referencescreen K (visual confirmation screen S) stored in the storage unit 90;the input control unit 76 then controls (selects) original screen M tobe displayed on the display apparatus 20 in accordance with therecognized contents of the inputted instruction.

In the first modification of the second embodiment, even in case thereis a deviation between operation screen T, as recognized by the user,and reference screen K due to the position of the user's finger being infront of or behind the position of reference screen K, a deviationcorrecting unit obtains (calculates) the position of the user's fingeron reference screen K and an input control unit can accurately recognizethe contents of the instruction made using the finger. It is noted thatother advantages of the present modification are the same or similar tothose of the second embodiment described above.

The positional data generating unit 74 b is configured according to thefirst modification of the second embodiment so as to acquire the XYcoordinates of a projection of a finger actually operated by the user onreference screen K as the positional data of the finger. However, it isalso possible to configure the positional data generating unit 74similar to the first and second embodiments so as to acquire the XYcoordinates of a projection of a finger actually operated by the user onreference screen K as the positional data of the finger.

Second Modification of the Second Embodiment

Next, a computer display device according to a second modification ofthe second embodiment of the present teachings will be described. FIG.23 is a schematic plan view of the computer display device according tothe second modification of the second embodiment of the presentteachings and is a diagram that will be used to explain a processperformed by a deviation correcting unit to convert an X coordinate ofpositional data (on operation screen T) into an X coordinate ofpositional data on reference screen K. FIG. 24 is a schematic right sideview of the computer display device and is a diagram that will be usedto explain a process performed by a deviation correcting unit to converta Y coordinate of positional data (on operation screen T) into a Ycoordinate of positional data on reference screen K. It will beunderstood that FIG. 23 generally corresponds to FIG. 14 and FIG. 24generally corresponds to FIG. 15, such that only differencestherebetween need be described.

The computer display device 1 e according to the second modification ofthe second embodiment differs from the computer display device 1 baccording to the first modification of the second embodiment in that thedisplay apparatus 20 d according to the modification of the firstembodiment is used as the display apparatus. In other words, as shown inFIGS. 23 and 24, the display apparatus 20 d includes a small projector21, an optical system 22, and a half mirror 23 embedded in a right-sidelens section 11 b of the eyeglass main body 10 d.

In the computer display device 1 e according to the presentmodification, the process for converting positional data using thedeviation correcting unit 77 b is performed in a manner similar to theabove-described first modification of the second embodiment. Moreover,while the half mirror 23 is embedded in the lens section 11 b in thepresent modification, the respective points (Cc, Mc, Ec, pc, Pc, and thelike) and the respective distances (W, H, L, α, and the like) aredefined exactly the same as in the first modification of the secondembodiment as shown in FIGS. 23 and 24.

Operational advantages of the computer display device 1 e according tothe present modification are the same or similar to those of theabove-described first modification of the second embodiment.

Third Embodiment

Next, a computer display device according to a third embodiment of thepresent teachings will be described with reference to FIG. 16.

As shown in FIG. 16, the computer display device 1 c according to thethird embodiment basically includes the same components as theabove-described first embodiment.

The computer display devices according to the first and thirdembodiments mainly differ in that the camera control unit 33 c includesan autofocus control unit 331, the reference data generating unit 75 cgenerates data that enables the position and the size in space to bespecified as data (reference data) concerning a visual confirmationscreen, and the control unit 70 c includes a distance determining unit78 c that determines, when a user performs an operation on the visualconfirmation screen S using a finger, whether or not the position of thefinger is separated (spaced) by an approximately constant distance orless from the plane representing (containing) the visual confirmationscreen S that is obtained using the reference data.

The autofocus control unit 311 may be the same as the autofocus controlunit according to the second embodiment and may be configured to controlthe camera unit 31 so that a subject (object) at a prescribed positionwithin the imaging range is automatically brought into focus. In thethird embodiment, the imaging apparatus 30 c has a large number of focalpoints so that any position within the imaging range can beautomatically brought into focus. Therefore, when the user is performingan operation on the visual confirmation screen S using a finger, theimaging apparatus 30 c is capable of automatically bringing the fingerinto focus and capturing one or more images in a state where the fingeris in focus. In addition, when capturing an image of the subject(object) that has been automatically brought into focus, the autofocuscontrol unit 331 calculates data concerning the distance to the capturedsubject (object). The calculated distance data is then associated withthe image data. In addition, the image data obtained by the imagingoperation performed by the imaging apparatus 30 c and the distance dataassociated with the image data are sent to the control unit 70 c. As wasthe case in the above-described embodiments, the autofocus system may beeither (i) an active system, in which a subject is irradiated withinfrared rays, ultrasonic waves, or the like and the distance isdetermined based on the period of time until reflected waves aredetected or based on an irradiation angle or (i) a passive system, suchas a phase difference detection system or a contrast detection system,that measures distance using an image captured by the lens of the cameraunit 31.

When the user performs operations using a finger at three prescribedpositions on the visual confirmation screen S, such as at three of thecorners of the visual confirmation screen S, the reference datagenerating unit 75 c generates data that enables the position and thesize in a three-dimensional space to be specified as the data concerningthe visual confirmation screen S by using the positional data of thefinger at each position as generated by the positional data generatingunit 74 based on: (i) the image data for which the operation determiningunit 73 has determined that the operation at each prescribed position isa prescribed operation by and (ii) the distance data associated with theimage data used when generating the positional data of the finger; thereference data generating unit 75 c then stores the generated data asreference data in the storage unit 90. Specifically, for each positiondescribed above, coordinate information (three-dimensional data) in anXYZ coordinate system can be constructed based on the positional data ofthe finger (two-dimensional positional data) and the distance data(one-dimensional positional data), and the XYZ coordinate information(three-dimensional data) at the three positions described above can beused as the reference data. In addition, an equation of the planerepresenting (containing) the visual confirmation screen S in the XYZcoordinate system can also be calculated using the reference data.Generally, the plane representing (containing) the visual confirmationscreen S, which is specified in this manner, need not necessarily beparallel to an XY plane. It is noted that, in the third embodiment, theplane corresponding (containing) to the visual confirmation screen Sobtained based on the reference data concerning the visual confirmationscreen S will be referred to as a “reference screen”.

When the user performs an operation on the visual confirmation screen Susing a finger and the imaging apparatus 30 c captures one or moreimages of a subject (finger) in focus, the distance determining unit 78c determines whether or not the subject (finger) exists within anapproximately constant distance (set in advance) from the plane(reference screen) corresponding to (containing) the visual confirmationscreen S based on: (i) the positional data of the finger as generated bythe positional data generating unit 74 based on the image data for whichthe operation determining unit 73 has determined the operation performedby the finger is a prescribed operation, (ii) the distance dataassociated with the image data used when generating the positional dataof the finger, and (iii) the plane (reference screen) corresponding to(containing) the visual confirmation screen S which is obtained based onthe (stored) reference data concerning the visual confirmation screen S.When determining whether or not the finger exists within anapproximately constant distance from the reference screen, theapproximately constant distance refers to a distance (or range ofdistances) at which it can be recognized or assumed (confirmed) that theuser is intending to perform a valid operation (manipulation) on thevisual confirmation screen S. In this case, the (range of the)approximately constant distance is set to, for example, approximately 5cm. Accordingly, when determining that the finger is separated (spaced)from the reference screen by more than the approximately constantdistance, the distance determining unit 78 c recognizes or determinesthat the finger is performing an operation at a position that issignificantly in front of or behind the position of the visualconfirmation screen S. On the other hand, if it is determined that thefinger is located within the approximately constant distance from thereference screen, the distance determining unit 78 c recognizes ordetermines that the user is performing a valid operation (manipulation)on the visual confirmation screen S.

When the user performs an operation on the visual confirmation screen Susing a finger and the distance determining unit 78 c determines thatthe finger is located within the approximately constant distance fromthe reference screen, the input control unit 76 c recognizes thecontents of the inputted instruction corresponding to the operationperformed by the finger based on: (i) the data concerning the contentsof the operation performed by the finger as obtained by a determinationmade by the operation determining unit 73, (ii) the positional data ofthe finger generated by the positional data generating unit 74 based onthe image data used to make the determination, (iii) the distance dataassociated with the image data used to make the determination, and (iv)the reference data concerning the visual confirmation screen S stored inthe storage unit 90; the distance determining unit 78 c then controls(selects, generates) the original screen M to be displayed on thedisplay apparatus 20 in accordance with the recognized contents of theinputted instruction.

Next, a process for setting the reference data in the computer displaydevice 1 c according to the third embodiment will be described withreference to the flow chart according to the first embodiment shown inFIG. 8.

The reference data setting processes according to the first and thirdembodiments differ in the processes (steps S2 and S3) performed by theimaging apparatus 30 c and the process (step S8) for generating thereference data that is performed by the reference data generating unit75 c. Therefore, only the different processes (steps) of the referencedata setting process according to the third embodiment will be describedwith reference to the flow chart shown in FIG. 8.

First, the display control unit 71 creates (generates) an originalscreen for setting (inputting) the reference data and displays theoriginal screen on the display apparatus 20 (S1). FIGS. 17A and B showexemplary diagrams of the original screen M for setting the referencedata that is displayed during the process of setting the reference data.In the example shown in FIG. 17A, the original screen M for setting thereference data is the character input screen 201 to which imagesrepresenting circles and numerals have been added at three prescribedpositions (i.e. at positions corresponding to three of the fourcorners). It is noted that, although images of a circle are displayed atthe three prescribed corners of the character input screen 201 in FIG.17A, images of a circle may instead be displayed at three prescribedcorners (of the four total corners) of the keyboard image 210 on thecharacter input screen 201 as shown in FIG. 17B. After the process ofstep S1, the control unit 70 c starts an imaging operation performed by,or together with, the imaging apparatus 30 c (S2). Upon viewing thecharacter input screen 201 to which images of the circles and numeralshave been added at the three prescribed positions as shown in FIG. 17Aas the visual confirmation screen S for setting the reference data, theuser performs a prescribed operation, such as a tapping operation usinga finger, on the respective numbered circles in an ascending numericalorder on the visual confirmation screen S for setting the referencedata. One or more images of the operation performed by the user (theuser's finger) is (are) captured by the imaging apparatus 30 c. At thispoint, the autofocus control unit 331 controls the camera unit 31 sothat a subject (object, i.e. the finger) within the imaging range isautomatically brought into focus and the imaging apparatus 30 c capturesone or more images of the subject in focus. In addition, when capturingone or more images of the subject that has been automatically broughtinto focus, the autofocus control unit 331 calculates data concerningthe distance to the captured subject and associates the calculateddistance data with the image data. The image data obtained by thisimaging operation is sent to the image processing unit 32 and the imageprocessing unit 32 performs prescribed image processing on the imagedata. Subsequently, the image data, which has been subjected to theimage processing, and the distance data associated with the image dataare sent to the control unit 70 c (S3).

In step S8, using the positional data of the finger at the threeprescribed positions generated by the positional data generating unit 74in step S7 and the distance data associated with the image data usedwhen generating the positional data of the finger, the reference datagenerating unit 75 c generates reference data concerning the visualconfirmation screen S currently being displayed and stores the referencedata in the storage unit 90.

Next, a character input process in the computer display device 1 caccording to the third embodiment will be described with reference toFIG. 18, which is a flow chart that explains the character inputprocedure according to the third embodiment. In this embodiment, it isassumed that the character input screen 200 is displayed in advance onthe display apparatus 20 as original screen M.

The user first issues a voice instruction (command) to input charactersvia the microphone unit 50 or issues (inputs) a manual instruction toperform character input using the touchpad unit 40 or the like. Uponreceiving the instruction, the control unit 70 c reads the characterinput program from the storage unit 90 and executes (initiates) thecharacter inputting process in accordance with a processing flow shownin FIG. 18. It is noted that the character input process (program) maybe configured or adapted to be automatically executed when the characterinput screen 200 is displayed on the display apparatus 20 as theoriginal screen M.

Then, the control unit 70 c determines whether or not reference dataconcerning the visual confirmation screen S that corresponds to originalscreen M, which is currently being displayed on the display apparatus20, is stored in the storage unit 90 (S121). When reference dataconcerning the visual confirmation screen S is not stored in the storageunit 90, the control unit 70 c reads the reference data setting programfrom the storage unit 90 and executes a reference data setting process(S122), as was described in the preceding embodiments. Subsequently, theprocess returns to step S121. It is noted that, although the presentembodiment (like the above-described embodiments) executes the referencedata setting process when reference data concerning the visualconfirmation screen S is not stored in the storage unit 90, even ifreference data concerning the visual confirmation screen S is stored inthe storage unit 90, the reference data setting process may be executedwhen an instruction is received from the user to once again generate thereference data.

When it is determined in step S121 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 c starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 c (S123). Using a finger, the userperforms a prescribed operation such as a tapping operation on thekeyboard image 210 of the character input screen 200, which is thevisual confirmation screen S. One or more images of the operationperformed by the user (the user's finger) is (are) captured by theimaging apparatus 30 c. At this point, the autofocus control unit 331controls the camera unit 31 so that a subject (object, e.g., a finger)within the imaging range is automatically brought into focus and theimaging apparatus 30 c captures one or more images of the subject infocus. In addition, when capturing the image(s) of the subject that hasbeen automatically brought into focus, the autofocus control unit 331calculates data concerning the distance to the captured subject andassociates the calculated distance data with the image data. The imagedata obtained by this imaging operation is sent to the image processingunit 32 and the image processing unit 32 performs prescribed imageprocessing on the image data. Subsequently, the image data, which hasbeen subjected to the image processing, and the distance data associatedwith the image data are sent to the control unit 70 c (S124).

Next, the image data extracting unit 72 determines whether or not thesubject is a finger based on the image data obtained by imagingperformed by the imaging apparatus 30 c and, if so, extracts image datain which the finger is present (S125). Subsequently, the operationdetermining unit 73 determines whether or not the operation performed bythe finger is a prescribed operation (in this case, a tapping operation)based on the image data extracted by the image data extracting unit 72.This determination is made within a prescribed (predetermined) period oftime (set in advance). In addition, when the operation performed by thefinger is a tapping operation, the operation determining unit 73determines that a character input operation has been successfully(normally) recognized, and when the operation performed by the finger isnot a tapping operation, the operation determining unit 73 determinesthat a character input operation has not been successfully (normally)recognized (S126). When it is determined that a character inputoperation has been successfully (normally) recognized, the operationdetermining unit 73 stores data concerning the contents of the operationperformed by the finger in the storage unit 90 and, at the same time,sends a signal informing the display control unit 71 that a characterinput operation has been successfully (normally) recognized. When thedisplay control unit 71 receives the signal, the display control unit 71adds an image, e.g., indicating a green lamp, to the original screen Msignifying that the character input operation has been successfully(normally) recognized and displays the original screen M on the displayapparatus 20 (S128). It is noted that, together with the imageindicating a green lamp or in place of the image indicating a greenlamp, the display control unit 71 may add images to the original screenM indicating one or more characters or graphics signifying that thecharacter input operation has been successfully (normally) recognized.Alternatively, as was described above, the control unit 70 c may cause aspecific notification sound to be emitted from the speaker unit 60together with, or in place of, displaying an image signifying that thecharacter input operation has been successfully (normally) recognized.

On the other hand, when the operation determining unit 73 determines instep S126 that a character input operation has not been successfully(normally) recognized within the prescribed (predetermined) period oftime (set in advance), the operation determining unit 73 sends a signalto that effect to the display control unit 71. At this point, forexample, if image data in which a finger is present is not sent from theimage data extracting unit 72 within the prescribed (predetermined)period of time (set in advance), the operation determining unit 73determines that a character input operation has not been successfully(normally) recognized. When the display control unit 71 receives thesignal, the display control unit 71 adds an image, e.g., indicating ared lamp, to the original screen M signifying that a character inputoperation has not been successfully (normally) recognized and displaysthe original screen M on the display apparatus 20 (S127). Subsequently,the process returns to step S133. It is noted that, at this point,together with the image indicating a red lamp or in place of the imageindicating a red lamp, the display control unit 71 may add images to theoriginal screen M indicating one or more characters or graphicssignifying that a character input operation has not been successfully(normally) recognized. Alternatively, as was described above, thecontrol unit 70 c may cause a specific notification sound to be emittedfrom the speaker unit 60 together with, or in place of, displaying animage signifying that the character input operation has not beensuccessfully (normally) recognized.

After the process of step S128, the positional data generating unit 74generates positional data of the finger (fingertip) in the imaging rangeof the imaging apparatus 30 c based on image data for which theoperation determining unit 73 has determined the operation performed bythe finger is a tapping operation (S129). The positional data of thefinger generated in this manner is stored in the storage unit 90.

Next, based on: (i) the positional data of the finger generated by thepositional data generating unit 74, (ii) the distance data associatedwith the image data used when generating the positional data of thefinger, and (iii) the reference data concerning the visual confirmationscreen S stored in the storage unit 90, the distance determining unit 78c determines whether or not the finger is within an approximatelyconstant distance set in advance from the plane (reference screen)corresponding to (containing) the visual confirmation screen S (S130).When the distance determining unit 78 c determines that the finger isseparated spaced from the reference screen by more than the(predetermined) approximately constant distance, the distancedetermining unit 78 c determines that the user is not performing a validoperation on the visual confirmation screen S and subsequently returnsto step S127. On the other hand, when the distance determining unit 78 cdetermines in step S130 that the finger is located within theapproximately constant distance from the reference screen, the distancedetermining unit 78 c recognizes or determines that the user isperforming a valid operation on the visual confirmation screen S andsubsequently returns to step S131.

In step S131, based on: (i) the data concerning the contents of theoperation performed by the finger determined and obtained by theoperation determining unit 73, (ii) the positional data of the finger asgenerated by the positional data generating unit 74, (iii) the distancedata associated with image data used to make the determination, and (iv)the reference data concerning the visual confirmation screen S stored inthe storage unit 90, the input control unit 76 c recognizes the contentsof the inputted instruction corresponding to the operation performed bythe finger. For example, when the user performs a tapping operationusing a finger with respect to a character key image in the keyboardimage 210, by checking which character key image region in the keyboardimage 210 to which position of the finger obtained from the positionaldata of the finger corresponds, the input control unit 76 c can specify(identify) the character key on which the current tapping operation hasbeen performed and recognize (determine) that the input of the specified(identified) character key has been instructed (requested, intended bythe user). Subsequently, the input control unit 76 c sends a signalconcerning the recognized contents of the inputted instruction to thedisplay control unit 71 and the display control unit 71 displaysoriginal screen M, in accordance with the contents of the inputtedinstruction, on the display apparatus 20 (S132).

After the process of step S132 or step S127, the control unit 70 cdetermines whether or not an instruction to end (terminate) thecharacter input process has been received from the user (S133). When aninstruction to end the character input process has been received, thecharacter input process is ended (terminated). On the other hand, whenan instruction to end the character input process has not been received,the process returns to step S123 and the character input process iscontinued. It is noted that the user may issue the instruction to endcharacter input process by voice, by a touch operation on the touchpadunit 40, or the like, as was repeatedly described in the precedingembodiments.

Next, a screen display process according to the third embodiment will bedescribed with reference to the flow chart of FIG. 19.

The user first issues (inputs) a voice instruction (command) to performa screen display operation via the microphone unit 50 or issues (inputs)a manual instruction to perform the screen display operation using thetouchpad unit 40, the visual confirmation screen S or the like. Uponreceiving the instruction, the control unit 70 c reads the screendisplay program from the storage unit 90 and executes (initiates) ascreen displaying process in accordance with the processing flow shownin FIG. 19. It is noted that the screen displaying process may beconfigured to be automatically executed when original screen M isdisplayed on the display apparatus 20.

Then, the control unit 70 c determines whether or not reference dataconcerning the visual confirmation screen S corresponding to theoriginal screen M currently being displayed on the display apparatus 20is stored in the storage unit 90 (S141). When reference data concerningthe visual confirmation screen S is not stored in the storage unit 90,the control unit 70 c reads the reference data setting program from thestorage unit 90 and executed a reference data setting process (S142), inthe same manner as was described in the preceding embodiments.Subsequently, the process returns to step S141. It is once again notedthat, although the present embodiment executes the reference datasetting process when reference data concerning the visual confirmationscreen S is not stored in the storage unit 90, even if reference dataconcerning the visual confirmation screen S is stored in the storageunit 90, the reference data setting process may be executed when aninstruction is received from the user to once again generate thereference data.

When it is determined in step S141 that reference data concerning thevisual confirmation screen S is stored in the storage unit 90, thecontrol unit 70 c starts an imaging operation performed by, or togetherwith, the imaging apparatus 30 c (S143).

The user then uses a finger to perform a prescribed operation, such as atapping operation, on the keyboard image 210 of the character inputscreen 200, which is the visual confirmation screen S. One or moreimages of the operation performed by the user (the user's finger) is(are) captured by the imaging apparatus 30 c. At this point, theautofocus control unit 331 controls the camera unit 31 so that a subject(object, e.g., a finger) within the imaging range is automaticallybrought into focus and the imaging apparatus 30 c captures one or moreimages of the subject in focus. In addition, when capturing the image(s)of the subject that has been automatically brought into focus, theautofocus control unit 331 calculates data concerning distance to thecaptured subject and associates the calculated distance data with theimage data. The image data obtained by this imaging operation is sent tothe image processing unit 32 and the image processing unit 32 performsprescribed image processing on the image data. Subsequently, the imagedata, which has been subjected to the image processing, and the distancedata associated with the image data are sent to the control unit 70 c(S144).

Next, the image data extracting unit 72 determines whether or not thesubject is a finger based on the image data obtained by imagingperformed by the imaging apparatus 30 c and, if so, extracts image datain which the finger is present (S145). Subsequently, the operationdetermining unit 73 determines the contents of the operation performedby the finger based on the image data extracted by the image dataextracting unit 72. In addition, the operation determining unit 73determines whether or not the operation performed by the finger has beensuccessfully (normally) recognized (S146). When it is determined that anoperation performed by the finger has been successfully (normally)recognized, the operation determining unit 73 stores data concerning thecontents of the operation performed by the finger in the storage unit 90and, at the same time, sends a signal informing the display control unit71 that the operation performed by the finger has been successfully(normally) recognized. When the display control unit 71 receives thesignal, the display control unit 71 adds an image, e.g., indicating agreen lamp, to original screen M signifying that the operation performedby the finger has been successfully (normally) recognized and displaysthe original screen M on the display apparatus 20 (S148). It is againnoted that, together with the image indicating a green lamp or in placeof the image indicating a green lamp, the display control unit 71 mayadd one or more images to original screen M indicating one or morecharacters or graphics signifying that the operation performed by thefinger has been successfully (normally) recognized. Alternatively, aswas described several times above, the control unit 70 may cause aspecific notification sound to be emitted from the speaker unit 60together with, or in place of, displaying an image signifying that theoperation performed by the finger has been successfully (normally)recognized.

On the other hand, when the operation determining unit 73 determines instep S146 that an operation performed by the finger has not beensuccessfully (normally) recognized, the operation determining unit 73sends a signal to that effect to the display control unit 71. At thispoint, for example, if image data in which a finger is present is notsent from the image data extracting unit 72 within the prescribed(predetermined) period of time (set in advance), the operationdetermining unit 73 determines that a character input operation has notbeen successfully (normally) recognized. When the display control unit71 receives the signal, the display control unit 71 adds an imageindicating a red lamp to original screen M signifying that a characterinput operation performed by the finger has not been successfully(normally) recognized and displays original screen M on the displayapparatus 20 (S147). Subsequently, the process returns to step S153. Itis again noted that, at this point, together with the image indicating ared lamp or in place of the image indicating a red lamp, the displaycontrol unit 71 may add images to original screen M indicating one ormore characters or graphics signifying that the operation performed bythe finger has not been successfully (normally) recognized.Alternatively, as was described above, the control unit 70 c may cause aspecific notification sound to be emitted from the speaker unit 60together with, or in place of, displaying an image signifying that theoperation performed by the finger has not been successfully (normally)recognized.

After the process of step S148, the positional data generating unit 74generates positional data of the finger (fingertip) in the imaging rangeof the imaging apparatus 30 c based on the image data representing thecontents of the operation performed by the finger as determined by theoperation determining unit 73 (S149). The positional data of the fingergenerated in this manner is stored in the storage unit 90.

Next, based on: (i) the positional data of the finger generated by thepositional data generating unit 74, (ii) the distance data associatedwith the image data used when generating the positional data of thefinger, and (iii) the reference data concerning the visual confirmationscreen S stored in the storage unit 90, the distance determining unit 78c determines whether or not the finger is within a predetermined,approximately-constant distance (set in advance) from the plane(reference screen) corresponding to (containing) the visual confirmationscreen S (S150). When the distance determining unit 78 c determines thatthe finger is separated (spaced) from the reference screen by more thanthe approximately constant distance, the distance determining unit 78 cdetermines that the user is not performing a valid operation on thevisual confirmation screen S and subsequently the process returns tostep S147. On the other hand, when the distance determining unit 78 cdetermines in step S150 that the finger is located within theapproximately constant distance from the reference screen, the distancedetermining unit 78 c recognizes or determines that the user isperforming a valid operation on the visual confirmation screen S andsubsequently the process returns to step S151.

In step S151, based on: (i) the data concerning to the contents of theoperation performed by the finger determined and obtained by theoperation determining unit 73, (ii) the positional data of the finger asgenerated by the positional data generating unit 74, (iii) the distancedata associated with the image data used to make the determination, and(iv) the reference data concerning the visual confirmation screen Sstored in the storage unit 90, the input control unit 76 c recognizesthe contents of the inputted instruction corresponding to the operationperformed by the finger. For example, when the user uses a finger toperform a double-tapping operation on the visual confirmation screen S,the input control unit 76 c specifies (identifies) that the currentoperation is a double-tapping operation and recognizes or determinesthat an instruction to enlarge (or reduce) original screen M has beenreceived. Subsequently, the input control unit 76 c sends a signalconcerning the recognized contents of the inputted instruction to thedisplay control unit 71 and the display control unit 71 displaysoriginal screen M, in accordance with the contents of the instruction,on the display apparatus 20 (S152).

After the process of step S152 or step S147, the control unit 70 c maydetermine whether or not to terminate the screen display process (S153)in the same manner as was described in the preceding embodiments.

Operational advantages of the computer display device according to thethird embodiment are similar to those of the computer display deviceaccording to the first embodiment. In particular, in the thirdembodiment, since the reference data generating unit generates datacapable of specifying the position and the size of the visualconfirmation screen S in space as reference data, even if the user is aperson having a habit of, when operating the visual confirmation screenS using fingers involving, for example, operating at positions on a nearside with respect to two left-side corners of the visual confirmationscreen S and operating at positions on a far side with respect to tworight-side corners of the visual confirmation screen S, reference dataconforming to such a habit can be generated.

Moreover, in the computer display device according to the thirdembodiment, the display apparatus 20 d according to the modification ofthe first embodiment may be used as the display apparatus. In this case,as shown in FIGS. 21A to 22B, the display apparatus 20 d includes asmall projector 21, an optical system 22, and a half mirror 23 embeddedin a right-side lens section 11 b of the eyeglass main body 10 d.

Other Embodiments

The present teachings are not limited to the above-described embodimentsand modifications; various changes can be made to the present teachingswithout departing from the spirit and scope of the invention.

For example, although the above-described embodiments and modificationsthereof all involve a user performing operations on a visualconfirmation screen using a finger, the user may instead or also performoperations on the visual confirmation screen for example, using an inputinstructing tool (writing utensil) such as a stylus or a pen.

In addition or in the alternative, in the above-described embodimentsand modifications, when performing touch operations on the visualconfirmation screen using a finger, the user may attach a prescribedmark to a fingertip, such as a simple graphic, symbol, or the like,e.g., a circle or a square. In such embodiments, the mark may beattached to a fingertip as, or in the form of, e.g., (i) an adhesivesticker inscribed with (depicting) the mark, (ii) a thimble or othercovering having the mark that is worn on the fingertip or (iii) a ringinscribed with the mark worn around the fingertip, in addition todrawing the mark directly on the fingertip. Moreover, if touchoperations are performed with a finger on the front side of the visualconfirmation screen as shown in FIGS. 6A and 7A, the mark may beattached to the nail side of the finger. On the other hand, if touchoperations are performed with a finger on the rear side of the visualconfirmation screen (in addition or in alternative to the front side) asshown in FIGS. 6B and 7B, the mark may be attached to both the nail sideof the finger and the fingertip or the pad (fingerprint) side of thefinger. It is noted that, if touch operations are performed with afinger only on the rear side of the visual confirmation screen, the markmay be attached only to the fingertip or to the pad side of the finger.In any of these embodiments, the image data extracting unit extractsimage data in which the mark is present as the image data in which thefinger is present. In addition, the operation determining unitdetermines the contents of the operation performed by a finger based onmovement of the mark and the positional data generating unit generatespositional data of the mark as the positional data of the finger. Inactual analysis and processing of image data, since a mark (such as asimple graphic or symbol) can be more easily and accurately recognizedthan a finger, the image (object) recognition precision can beincreased.

In addition or in the alternative, although the above-described first,second, and third embodiments as well as the first modification of thesecond embodiment utilize a display apparatus that is provided separatefrom a lens of the eyeglass, the present teachings are not limitedthereto and the display apparatus may be integrally configured with alens.

In addition or in the alternative, although the above-described first,second, and third embodiments and the first modification of the secondembodiment utilize a display apparatus that includes a small projectorhaving a display device, an optical system, and a half mirror, whereinthe half mirror is embedded in a prism arranged (disposed) in front of alens section of an eyeglass main body, the half mirror may be integratedwith the prism or may be affixed to the prism. In addition or in thealternative, although the above-described modification of the firstembodiment and in the second modification of the second embodimentutilize a display apparatus that includes a small projector having adisplay device, an optical system, and a half mirror, wherein the halfmirror is embedded in a lens section of an eyeglass main body, the halfmirror may be affixed to the lens section or may be integrated with thelens section.

In addition or in the alternative, although the above-describedembodiments and modifications utilize a display apparatus that includesa small projector having a display device, an optical system, and a halfmirror, a holographic sheet or a holographic film that reflects a partof light (video) may be used in place of the half mirror. In suchembodiments, the holographic sheet or the holographic film is affixed toa part of or all of a lens section of an eyeglass main body, is embeddedin the lens section of the eyeglass main body, or is integrated with thelens section of the eyeglass main body. Accordingly, an image or a videofrom the small projector is projected onto the holographic sheet or theholographic film via the optical system and the user can view the imageor the video reflected by the holographic sheet or the holographic film.

In addition or in the alternative, although the above-describedembodiments and modifications utilize one original screen that isdisplayed on a display apparatus, the present teachings are not limitedthereto and two original screens may be displayed on the displayapparatus. In such embodiments, the user can view two visualconfirmation screens that appear as though they are floating in midair.For example, if the keyboard image on the character input screen isdivided into two such that the keyboard image is constituted by aright-side keyboard image and a left-side keyboard image, the user canrecognize the right-side keyboard image and the left-side keyboard imageas though they are floating in midair and can input characters usingfingers of both hands.

In addition or in the alternative, in any of the above-describedembodiments and modifications, two display apparatuses may be providedand a different original screen may be displayed in (by) each displayapparatus. For example, by arranging one of the display apparatuses infront of the lens section for the right eye and the other displayapparatus in front of the lens section for the left eye, the user canrecognize (view) one of the visual confirmation screens with the righteye and recognize (view) the other visual confirmation screen with theleft eye. In this case, two imaging apparatuses may be provided so thateach imaging apparatus captures an image of a finger performing anoperation on each visual confirmation screen.

In addition or in the alternative, although the above-describedembodiments and modifications utilize an XY coordinate system that isset within the imaging range of the imaging apparatus and the origin ofthe XY coordinate system is a bottom left point in the imaging range,the origin of the XY coordinate system may be set to any position in theimaging range. For example, the origin of the XY coordinate system maybe a top left point in the imaging range.

In addition or in the alternative, in the above-described secondembodiment, when the image data extracting unit determines whether ornot a subject is separated from the imaging apparatus in the Z axisdirection by an approximately constant distance, the approximatelyconstant distance is set to a distance within a range of approximately40 cm±5 cm from the imaging apparatus. However, the present teachingsare not limited thereto and the approximately constant distance may beset to, for example, 30 cm±3 cm or 50 cm±6 cm or any valuestherebetween. Alternatively, a configuration may be adopted whichenables the user to freely change (set) the approximately constantdistance. In a similar manner, in the third embodiment, when thedistance determining unit determines whether or not a finger is locatedwithin an approximately constant distance from the reference screen, theapproximately constant distance is set to approximately 5 cm. However,the present teachings are not limited thereto and the approximatelyconstant distance may be set to, for example, 3 cm or 6 mm or any valuestherebetween. Alternatively, a configuration may be adopted whichenables the user to freely change (set) the approximately constantdistance.

In addition or in the alternative, in the third embodiment, when a userperforms operations using a finger at three prescribed positions on avisual confirmation screen such as three of the corners of the visualconfirmation screen, the reference data generating unit obtains XYZcoordinate information at the three positions using the positional dataof the finger at each position and distance data associated with imagedata used when generating the positional data of the finger, and thereference data generating unit utilizes the obtained XYZ coordinateinformation at the three positions as the reference data. However, thepresent teachings are not limited thereto. For example, when the userperforms operations using a finger at at least three prescribedpositions on the visual confirmation screen, the reference datagenerating unit may obtain XYZ coordinate information at the at leastthree positions using the positional data of the finger at each positionand the distance data sent together with image data used when generatingthe positional data of the finger, and may create (generate) thereference data based on the obtained XYZ coordinate information at eachposition. For example, when the user performs operations using a fingerat the four corners of the visual confirmation screen, the referencedata generating unit may obtain XYZ coordinate information at the fourpositions using the positional data of the finger at each position andthe distance data associated with the image data used when generatingthe positional data of the finger, and may generate reference data basedon the obtained XYZ coordinate information at the four positions. It isnoted that, in such embodiments, if the four positions are not on thesame plane when obtaining the XYZ coordinate information at the fourpositions, the reference data generating unit preferably obtains(determines) a plane with the smallest deviation from the four positionsusing a prescribed algorithm and generates reference data based on theobtained plane.

In addition or in the alternative, in the third embodiment, when a userperforms operations using a finger at three prescribed positions on thevisual confirmation screen, the reference data generating unitconstructs coordinate information (three-dimensional data) in an XYZcoordinate system based on the positional data of the finger(two-dimensional positional data) and the distance data (one-dimensionalpositional data) at each position of the finger, and uses thethree-dimensional data at the three positions as the reference data.However, for example, if two camera units are provided andthree-dimensional positional data of a subject (object) can be obtainedbased on the image data (stereoscopic image data) obtained by capturingan image of the subject with each camera unit, the reference datagenerating unit may use three-dimensional positional data of eachposition of the finger that is obtained when the user performs anoperation using the finger at three prescribed positions on the visualconfirmation screen as the reference data.

In addition or in the alternative, although the above-described secondand third embodiments utilize an autofocus control unit that controls animaging apparatus so that a subject (object) is automatically broughtinto focus, the autofocus control unit may control the imaging apparatusso that only a subject (object) located within a constant distance fromthe imaging apparatus in the Z axis distance is automatically broughtinto focus. For example, only a subject (object) within one meter fromthe imaging apparatus may be brought into focus and a subject (object)separated (spaced) from the imaging apparatus by more than one meter isnot brought into focus. Accordingly, the process for controlling theimaging apparatus so as to bring a subject (object) into focus and theprocess for extracting image data in which the finger is present fromthe image data captured by the imaging apparatus can be facilitated.

In addition or in the alternative, in the above-described embodiments,the in-focus range is limited to a narrow range during image capture bythe imaging apparatus when performing operations for reference datasetting, character input, and screen display. However, the presentteachings are not limited thereto and, for example, the in-focus rangemay be limited to a constant range when performing normal cameraphotography using the imaging apparatus (digital camera) of the computerdisplay device according to the present teachings. For example, whenperforming normal camera photography, a subject (object) that isseparated (spaced) from the imaging apparatus by more than one meter isnot brought into focus. In addition, when the in-focus range is limitedto a constant range such as when performing an operation for setting thereference data as described above or when performing normal cameraphotography, notifying means for notifying a third party that thein-focus range is being limited is preferably provided. As the notifyingmeans, for example, notification may be performed by turning on a lightsuch as an LED provided on the eyeglass main body or by displaying asign to that effect on the eyeglass main body, in both cases so thatthird parties (i.e. people around the person wearing the presentcomputer display device). In this regard, it is noted that knowncomputer display devices mounted on eyeglasses s create the risk ofinfringing on the privacy of people nearby when photographing using theimaging apparatus, which has created controversy concerning the use ofsuch devices. However, with the present teachings, by not bringing intofocus a subject that is separated (spaced) from the imaging apparatusby, for example, more than one meter and by providing notifying meansfor notifying third parties (nearby people) that the in-focus range isbeing limited to a (short) constant range, the issue of privacy (whichhas been a major concern for conventional computer display devicesmounted on eyeglasses) can be solved. Furthermore, the imaging apparatusmay be an imaging apparatus capable of limiting the in-focus range to a(short) constant range regardless of the state of the computer displaydevice according to the present teachings. In addition, when limitingthe in-focus range to a constant range regardless of the state of thecomputer display device, the notifying means desirably notifies thirdparties (nearby people) that the in-focus range is being limited to aconstant range even when the user is wearing the computer display deviceand performing an operation other than an operation for settingreference data described above and an operation for normal cameraphotography or even when the computer display device is in a standbystate. Furthermore, for example, when the imaging apparatus is notoperating or when the computer display device is not operating, anotification that the imaging apparatus or the computer display deviceis not operating is desirably made by the notifying means. Thenotification by the notifying means at this point may be similar to thenotification described above for notifying third parties (nearby people)that the in-focus range is being limited to a constant range or adifferent notification may be used by, for example, changing the colorof a light such as an LED.

In addition or in the alternative, in the second embodiment or the thirdembodiment described above, a configuration may be adopted in which,during a character input process or a screen display process, theimaging apparatus recognizes a finger as the subject (object) andautomatically and continuously brings the recognized finger into focus.

In addition or in the alternative, in the above-described embodimentsand modifications, the storage unit may store, as one or more originalscreens, e.g., (i) an operation screen for a mobile terminal such as amobile phone, a smartphone, a tablet, and a laptop computer, (ii) anoperation screen for an information terminal such as a personalcomputer, a remote controller screen for a household electric appliancesuch as a lighting fixture, a television set, an air conditioner, and asecurity system, or (iii) a control panel screen for electric equipmentsuch as a car stereo, a car navigation system, and/or an airconditioner. In such embodiments, the user causes the operation screenor the like described above to be displayed on the display apparatus asthe original screen by making a voice command or by performing a manualoperation using a touchpad, the visual confirmation screen or the like.Then, when performing an operation using a finger on the visualconfirmation screen that corresponds to the original screen, a controlunit (remote control unit) generates a command signal (command)indicating the contents of the operation and wirelessly transmits thegenerated command signal to the mobile terminal (mobile computingdevice) or the like described above via a communicating unit, e.g.,wirelessly. In this manner, the mobile terminal (mobile computingdevice) or the like described above can be remotely operated. In otherwords, the computer display device according to the present teachingscan also be used as a remote controller for a remotely-controllableapparatus such as an air conditioner. FIG. 20 is a diagram showing anembodiment in which a screen (remote controller screen) corresponding toan operating unit of a remote controller of an air conditioner is usedas the original screen and the user performs an operation on a visualconfirmation screen that corresponds to the remote controller screen. Asshown in FIG. 20, when the user operates, using a finger, atemperature-setting button on the visual confirmation screen so as tolower the temperature, the control unit generates a command for loweringthe set temperature and transmits the command by infrared communicationto the air conditioner via the communicating unit. In this manner, theuser can conveniently lower the set temperature of the air conditioner.

In addition or in the alternative, in all of the above-describedembodiments and modifications, the computer display device according tothe present teachings is mounted on the head of a user in the form ofeyeglasses. However, computer display devices according to the presentteachings are not limited to computing devices that are used while beinghooked around the ears as long as a display apparatus, an imagingapparatus, and the like are provided, and may be any kind of wearablecomputer device worn on the back of the head, the front of the head, orthe top of the head such as a headset type terminal or a head-mounteddisplay (HMD) type terminal.

In addition or in the alternative, the computer display device accordingto the present teachings may be an eyeglass-mounted computing device. Inthis case, by mounting the computer display device according to thepresent teachings to an eyeglass frame or one of the temples to be wornby the user, the computer display device according to the presentteachings can be mounted or worn on the head.

As was described above, with the computer display devices according tothe present teachings, when a user performs an operation on a visualconfirmation screen using a finger, a mark attached to the finger or aninput instructing tool (collectively, “an indicator”), an input controlunit recognizes the contents of the inputted instruction, whichcorresponds to the operation performed by the finger or the inputinstructing tool (indicator), based on: (i) data concerning the contentsof the operation performed by the finger or the input instructing tool(indicator) as obtained by a determination made by an operationdetermining unit, (ii) positional data of the finger or the inputinstructing tool (indicator) generated by a positional data generatingunit, and (iii) reference data concerning the visual confirmation screenthat is stored in a storage unit. The input control unit then controls(selects, generates) an original screen to be displayed on a displayapparatus in accordance with the recognized contents of the inputtedinstruction. Therefore, by using the wearable computer display deviceaccording to the present teachings, the user can easily and accuratelyperform character input operations and various screen operationsincluding enlargement and reduction by operating a visual confirmationscreen in a manner similar to a conventional smartphone or a tabletcomputer. Accordingly, the present teachings can be applied to computerdisplay devices that are worn by a user in a similar manner toeyeglasses.

Representative, non-limiting examples of the present invention weredescribed above in detail with reference to the attached drawings. Thisdetailed description is merely intended to teach a person of skill inthe art further details for practicing preferred aspects of the presentteachings and is not intended to limit the scope of the invention.Furthermore, each of the additional features and teachings disclosedabove may be utilized separately or in conjunction with other featuresand teachings to provide improved wearable computers, such as computerdisplay devices, eyeglass-type computer displays, “smartglasses”, etc.

Moreover, combinations of features and steps disclosed in the abovedetailed description may not be necessary to practice the invention inthe broadest sense, and are instead taught merely to particularlydescribe representative examples of the invention. Furthermore, variousfeatures of the above-described representative examples, as well as thevarious independent and dependent claims below, may be combined in waysthat are not specifically and explicitly enumerated in order to provideadditional useful embodiments of the present teachings.

All features disclosed in the description and/or the claims are intendedto be disclosed separately and independently from each other for thepurpose of original written disclosure, as well as for the purpose ofrestricting the claimed subject matter, independent of the compositionsof the features in the embodiments and/or the claims. In addition, allvalue ranges or indications of groups of entities are intended todisclose every possible intermediate value or intermediate entity forthe purpose of original written disclosure, as well as for the purposeof restricting the claimed subject matter.

Although some aspects of the present invention have been described inthe context of a device or apparatus, it is to be understood that theseaspects also represent a description of a corresponding method, so thata block or a component of a device or apparatus is also understood as acorresponding method step or as a feature of a method step. In ananalogous manner, aspects which have been described in the context of oras a method step also represent a description of a corresponding blockor detail or feature of a corresponding device.

Depending on certain implementation requirements, exemplary embodimentsof the invention, such as the control unit and/or functional blocksthereof may be implemented in hardware and/or in software. Theimplementation can be performed using a digital storage medium, forexample a ROM, a RAM, a PROM, an EPROM, an EEPROM or a flash memory, onwhich electronically readable control signals are stored, which interactor can interact with a programmable hardware component such that therespective method is performed.

A programmable hardware component can be formed by one or more of aprocessor, a computer processor (CPU=central processing unit), agraphics processor (GPU=graphics processing unit), a computer, acomputer system, an application-specific integrated circuit (ASIC), anintegrated circuit (IC), a system-on-a-chip (SOC), a programmable logicelement, or a field programmable gate array (FGPA) including amicroprocessor.

The digital storage medium can therefore be machine- or computerreadable. Some exemplary embodiments thus comprise a data carrier ornon-transient computer readable medium which includes electronicallyreadable control signals which are capable of interacting with aprogrammable computer system or a programmable hardware component suchthat one of the methods described herein is performed. An exemplaryembodiment is thus a data carrier (or a digital storage medium or anon-transient computer-readable medium) on which the program(s) forperforming one of the methods described herein is (are) recorded.

In general, exemplary embodiments of the present invention areimplemented as a program, firmware, computer program, or computerprogram product including a program, or as data, wherein the programcode or the data is operative to perform one of the methods if theprogram runs on a processor (e.g., a microprocessor) or otherprogrammable hardware component. The program code or the data can forexample also be stored on a machine-readable carrier or data carrier.The program code or the data can be, among other things, source code,machine code, bytecode or another intermediate code.

A further exemplary embodiment is a data stream, a signal sequence, or asequence of signals which represents the program for performing one ofthe methods described herein. The data stream, the signal sequence, orthe sequence of signals can for example be configured to be transferredvia a data communications connection, for example via the Internet oranother network. Exemplary embodiments are thus also signal sequenceswhich represent data, which are intended for transmission via a networkor a data communications connection, wherein the data represent theprogram.

A program according to an exemplary embodiment can implement one of themethods during its performance, for example, such that the program readsstorage locations or writes one or more data elements into these storagelocations, wherein switching operations or other operations are inducedin transistor structures, in amplifier structures, or in otherelectrical, optical, magnetic components, or components based on anotherfunctional principle. Correspondingly, data, values, sensor values, orother program information can be captured, determined, or measured byreading a storage location. By reading one or more storage locations, aprogram can therefore capture, determine or measure sizes, values,variable, and other information, as well as cause, induce, or perform anaction by writing in one or more storage locations, as well as controlother apparatuses, machines, and components, and thus for example alsoperform complex processes using displays, projectors, etc.

In the present disclosure, the terms “eyeglasses”, “eyeglass frame” and“eyeglass main body” have been utilized to describe all elements of theeyeglasses (spectacles). However, it is noted that eyeglasses generallyinclude a frame, which holds right and left lenses, and two temples,which are identical long and narrow arms that attach to the ends of theframe, e.g., with or without hinges, and are designed to extend over theears to keep eyeglasses in place on the user's head. A bridge may beprovided in the frame between the two lenses. Furthermore, nose pads maybe attached to the frame or directly to the lens and are designed torest on the user's nose. Instead of a frame, the eyeglasses may be“frameless”, in which case the lenses are connected directly to thebridge and the temples. The various components disclosed above, such asthe imaging apparatus, display and hardware, may be attached to any oneof these components, but the display is preferably attached to the frameor one of the lens, and the hardware, including e.g., the imagingapparatus (camera) and a touchpad, is preferably attached to thetemples.

An additional embodiment of the present teachings is a device configuredto be worn on a head of a user and used together with an indicatorselected from the group consisting of a finger of the user, a markattached to the finger, and an input instructing tool. The devicepreferably comprises an eyeglass having a frame and two temples attachedthereto. A display is mounted on the eyeglass, e.g., on the frame, andis configured to generate a virtual screen viewable by the user. Acamera is mounted on the eyeglass, e.g. on one of the temples, and isconfigured to capture images in front of the user. A central processingunit is mounted on the eyeglass, e.g., on one of the temples, andcomprises a microprocessor in communication with a memory. The memorystores one or more microprocessor-readable instructions (programs) that,when executed by the microprocessor:

-   -   cause the camera to capture images of the indicator performing        an operation on the virtual screen;    -   recognize the indicator on the captured images and generate        indicator image data;    -   determine the operation being performed by the indicator using        the indicator image data, e.g., by tracking movement of the        indicator;    -   generate positional data of the indicator in an imaging range of        the camera using the indicator image data;    -   generate reference data concerning the virtual screen by        detecting a reference data setting operation performed by the        indicator at one or more predetermined positions on the virtual        screen using said positional data;    -   store the reference data in the memory;    -   cause the display to generate a virtual screen in accordance        with the determined operation, the generated positional data and        the stored reference data.

EXPLANATION OF THE REFERENCE NUMERALS

1, 1 a, 1 b, 1 c, 1 d, 1 e Computer display device

10, 10 d Eyeglass main body

11, 11 a, 11 b Lens section

20, 20 d Display apparatus

21 Small projector

22 Optical system

23 Half mirror

30, 30 a, 30 c Imaging apparatus

31 Camera unit

32 Image processing unit

33, 33 a, 33 c Camera control unit

331 Autofocus control unit

40 Touchpad unit

50 Microphone unit

60 Speaker unit

70, 70 a, 70 b, 70 c Control unit

71 Display control unit

72, 72 a Image data extracting unit

73 Operation determining unit

74, 74 b Positional data generating unit

75, 75 c Reference data generating unit

76, 76 c Input control unit

77 b Deviation correcting unit

78 c Distance determining unit

80 Communicating unit

90 Storage unit

200 Character input screen

201 Character input screen (original screen for setting reference data)

210 Keyboard image

220 Display area

221 Search screen

2211 Keyword input section

2212 Search result display section

1-4.(canceled)
 5. A device configured to be like eyeglasses by a user,the device comprising: an eyeglass main body; a display apparatusprovided on the eyeglass main body and configured to display an originalscreen corresponding to a visual confirmation screen that appears to auser to be floating in midair; an imaging apparatus provided on theeyeglass main body, and including an autofocus control unit configuredto automatically bring a subject into focus, the imaging apparatus beingconfigured to calculate, when an image of the subject that has beenautomatically brought into focus by the autofocus control unit iscaptured, data concerning the distance to the captured subject and tooutput the calculated distance data together with image data obtained bythe imaging; an image data extracting unit configured to: (i) determine,when the image of the subject that has been automatically brought intofocus is captured by the imaging apparatus, whether or not the subjectis a finger or an input instructing tool based on the image data and toextract a subset of the image data in which the finger or the inputinstructing tool is present; an operation determining unit configured todetermine, when the imaging apparatus captures the image of the fingeror the input instructing tool used by the user to perform the operationon the visual confirmation screen, contents of the operation performedby the finger or the input instructing tool, based on the subset of theimage data extracted by the image data extracting unit; a positionaldata generating unit configured to generate, when the imaging apparatuscaptures the image of the finger or of the input instructing tool usedby the user to perform the operation on the visual confirmation screen,positional data of the finger or of the input instructing tool in animaging range of the imaging apparatus [[e]], based on the image datafor which the operation determining unit has determined the operationperformed by the finger or the input instructing tool is a prescribedoperation; a reference data generating unit configured to generate, whenthe user performs an operation using the finger or the input instructingtool at at least three predetermined positions on the visualconfirmation screen, data specifying a reference screen that correspondsto the visual confirmation screen in a three-dimensional space using (i)at least three pieces of positional data of the finger or of the inputinstructing tool generated by the positional data generating unit basedon the image data for which the operation determining unit hasdetermined the operation at each of the prescribed positions is theprescribed operation, and (ii) distance data sent together with theimage data used when generating the pieces of positional data, and tostore the generated data specifying the reference screen as referencedata in a storage unit; a distance determining unit configured todetermine, when the image of the subject that has been automaticallybrought into focus is captured by the imaging apparatus upon the userperforming an operation on the visual confirmation screen using a fingeror an input instructing tool after the reference data has been generatedby the reference data generating unit, whether or not the finger or theinput instructing tool is positioned within a range of a predetermined,approximately-constant distance from the reference screen based on: (i)the positional data of the finger or of the input instructing toolgenerated by the positional data generating unit based on the image datafor which the operation determining unit has determined the operationperformed by the finger or the input instructing tool is the prescribedoperation, (ii) the distance data associated with the image data usedwhen generating the positional data of the finger or of the inputinstructing tool, and (iii) the reference data that specifies thereference screen corresponding to the visual confirmation screen andwhich is stored in the storage unit; and an input control unitconfigured to determine, when the user performs the operation using thefinger or the input instructing tool on the visual confirmation screenand the distance determining unit has determined that the finger or theinput instructing tool is located within the range of the predetermined,approximately-constant distance from the reference screen, contents ofan inputted instruction corresponding to the operation performed by thefinger or the input instructing tool based on: (i) the data concerningthe contents of the operation performed by the finger or the inputinstructing tool determined by the operation determining unit, (ii) thepositional data of the finger or of the input instructing tool generatedby the positional data generating unit based on the image data used tomake the determination, (iii) the distance data associated with theimage data used to make the determination, and (iv) the reference datathat specifies the reference screen corresponding to the visualconfirmation screen and which is stored in the storage unit, and tocontrol the original screen to be displayed on the display apparatus inaccordance with the determined contents of the inputted instruction.6-14. (canceled)
 15. The device according to claim 5, wherein theoriginal screen displayed on the display apparatus is a character inputscreen including a keyboard image and a display area for displayinginputted characters.
 16. The device according to claim 5, wherein theoriginal screen displayed on the display apparatus includes a screencorresponding to an operating unit of a remote controller for aremotely-controllable apparatus, and the device further comprises aremote control unit configured to generate, when an operation isperformed on the screen corresponding to the operating unit of theremote controller, a command signal representative of the contents ofthe operation and to wirelessly transmit the generated command signal tothe remotely-controllable apparatus.
 17. The device according to claim5, wherein the imaging apparatus is configured to limit an in-focusrange to a constant range, and the device further comprises notifyingmeans for notifying a third party of the fact that the in-focus range isbeing limited to the constant range.
 18. The device according to claim5, wherein the display apparatus includes a projector having a displaydevice, an optical system, and a half mirror onto which the originalscreen displayed on the display device is projected via the opticalsystem, and the half mirror is embedded in or integrated with a prismarranged in front of a lens section of the eyeglass main body or isaffixed to a prism arranged in front of the lens section of the eyeglassmain body.
 19. The device according to claim 5, wherein the displayapparatus includes a projector having a display device, an opticalsystem, and a half mirror onto which the original screen displayed onthe display device is projected via the optical system, and the halfmirror is affixed to a lens section of the eyeglass main body or isembedded in or integrated with the lens section of the eyeglass mainbody.
 20. The device according to claim 5, wherein the display apparatusincludes a projector having a display device, an optical system, and aholographic sheet or a holographic film onto which the original screendisplayed on the display device is projected via the optical system, andthe holographic sheet or the holographic film is affixed to a lenssection of the eyeglass main body or is embedded in or integrated withthe lens section of the eyeglass main body.
 21. The device according toclaim 5, wherein the operation determining unit is configured todetermine contents of an operation performed by a finger based onmovement of a predetermined mark attached to a fingertip of the finger,and the positional data generating unit is configured to generatepositional data of the mark as the positional data of the finger. 22.The device according to claim 5, wherein the device is configured to beused by the user while being mounted on any portion of the head otherthan the ears instead of being worn like eyeglasses.
 23. The deviceaccording to claim 5, wherein the eyeglass main body comprises a frameand two temples, the display apparatus being mounted on the frame andthe imaging apparatus being mounted on one of the temples.